Guide to backup and restore big tables in DynamoDB completely

To be brief, in Foreks, we are depending on real-time data with no tolerance of high-latencies. The lower the latency, the better results we can provide to our customers/clients. After Amazon launched the Frankfurt region (eu-central-1) we decided to move from Ireland (eu-west-1) to Frankfurt. It’s much lower latency when you ping both regions from Istanbul.

We were looking for a practical and easy solution to the issue of ours but after trying many solutions (like Data Pipeline) and getting tired of their complexity, we tend to lean on some open-source projects to do our task.

After a couple researches, found this little awesome batch tool working with boto: https://github.com/bchew/dynamodump

We gave it a shot and it worked nearly-perfectly.

Read more

Radial Basis Function Neural Network Scala Implementation

 

In this post, I am going to demonstrate a two-step Scala implementation of Radial Basis Function Neural Network (RBFNetwork): (unsupervised) k-means clustering first, and (supervised) gradient descent second. This two-step implementation is fast and efficient compared to Multilayer Perceptron while providing good predictive performance. This is because unsupervised learning at the first step provides information about data distribution so that the second step can have an intuition about the data and fine-tune the model.

For us, most of the Machine Learning models that we use today are just black box approaches that we take from some library. It is good to have this ability for faster development; however, it would be nicer to know internal dynamics of their implementations. Therefore, I am hoping that this post will be simple enough to provide you that information.

Read more

OpenVPN-AccessServer on AWS

OpenVPN-AccessServer on AWS

Have you ever need to access your servers in virtual private cloud environment outside from “safe-zones”? Moreover, does your team willing to login on those machines from almost anywhere at the least appropriate times? And all this requires Volvo-Class security? Well, we got you covered.

Since this article focus on AWS, our challenge comprehends Two major concepts.

  • Enabling the content of landing pages via S3 endpoint in addition to IP white listing.
  • Making sure no one routes internet traffic through the cloud network yet without losing connectivity.

Before telling more, let’s assume a basic scenario; you’ve launched a t2.nano instance & started an openvpn server (with docker or not) and you created a security rule which allows your openvpn instance’s Elastic IP address to any machine in the vpc as well as your S3 buckets. Sounds great but not very ideal. The thing is, AWS doesn’t cost a penny for the network traffic inside your VPC. So from going one EC2 to another you have to reach it over internet. That means both cost and speed issues. Same rule applies for your buckets surely. Not even mentioning that you are also providing kind of an open source ZenMate solution to all your users because whatever they download or upload goes through your network and your very own encrypted connection. And that is something no system administrator wants to be responsible for.

So what is the exact solution? Obviously it isn’t to block outgoing traffic for 0.0.0.0/0 and I can assure you, we have tried countless approaches from config files to kernel parameters. Meet the most elegant way now, OpenVPN-AS

Read more

Gradle 3.1: Composite Builds, Here you Go!

Gradle 3.1: Composite Builds, Here you Go!

Tired of having your changing dependencies slow you down? Gradle 3.1’s composite builds provide you with a process of changing and refreshing your dependencies.

Our journey with Gradle started one and half years ago with this presentation. Everybody in the company just loved its expressive and easy-to-understand structure and realized that a plugin model was the wrong level of abstraction. Instead, language-based approaches were the right one in terms of their flexibility for the long term.

So it didn’t take long for almost 50 members of the development team to change the whole build infrastructure with Gradle. Gradle doesn’t just throw away the foundation that other build tools brought. Instead, it builds up easily and more powerfully on top of others while remaining 100% compatible with them. Therefore, this made Gradle not an alternative but an upgrade for us.

While we were migrating projects, simple pom.xml projects required only a few steps, such as; “gradle init,” but others needed more effort. Everything was good with Gradle except for one thing — “composite builds.”

And finally, our most-requested feature came out with Gradle 3.1, and now we love Gradle more than ever.

Read more

High Performance Streaming Data Processing Implementation and Analysis

High Performance Streaming Data Processing Implementation and Analysis

Introduction

This article is more like a proof of concept of an implementation of microservice development, based on a real world implementation of a specific project build on top of integration of two stock markets, Nasdaq and BIST. The project requires different type of approaches and know-how which can be grouped into two categories, stream data api implementation and data analysis. Both sides have different challenges as they have different requirements.
Read more

Theme, User Story, Epic

Theme, User Story, Epic

After Scrum methodology settled into the organizational culture, there are some issues that cause confusion and communication problems. First of all, we should not forget that the aim of all these descriptions are only for creating a common language in the organization. That is why, if you already have the common terminology for these definitions, it is alright to keep going with them. In this article I address the well accepted description of these concepts.

Just before defining the three main term, I want to emphasise two basic concepts;

Task & Feature

In real Scrum lifecycle, while creating Product Backlog, requirements are collected by Product Owner while asking the question of “What”. Each answer of this question is handled as Feature and associated with each other. This correlated packages of Features generate Wish List of the project.

Read more

Docker Containers With Gradle Application Plugin

After trying several different approaches, we came up with what we think is the most elegant way of integrating Docker into our build tool Gradle.

Docker and Gradle have been around for a while, and there are many tutorials, blog posts, etc. related to best practices. After trying several different approaches, we came up with what we think is an elegant way of integrating Docker with our build tool Gradle. What follows is a simple and elegant integration of the two technologies.

First, let’s simply start with why we chose and how we can use Gradle’s Application plugin without getting into Docker yet. The Application plugin works hand in hand with Groovy, Scala, and Java plugins to create an executable JVM application. Using the Application plugin itself also implies application of the Distribution plugin. So, when it comes to deciding what plugin to use for making executables, the Application plugin is the most official way of doing things.

Read more

Getting Started Deep Learning Revolution with Java

AI and deep learning are transforming the way we understand software, making computers more intelligent than we could even imagine just a decade ago. It is the technology behind self-driven cars, intelligent personal assistant computers, and decision support systems. Deep learning algorithms are being used across a broad range of industries. As the fundamental driver of AI, being able to tackle deep learning with Java is going to be a vital and valuable skill, not only within the tech world, but also for the wider global economy that depends upon knowledge and insight for growth and success.

Read more

Ruby on Rails Development Life-cycle on Docker Containers

Before the containers have been integrated with the applications, the RoR application’s deployments are managed either manually or maintaining more handy tool such as Capistrano. Either way, there are a couple of required procedures needs to be applied on every new change set of the source code wanted to be deployed as a version.

Administration point of view, RoR applications are threated as file based applications, similar to PHP.  Unlike Java or Go, there is not one binary/archived deployable artifact. Therefore, every single change set contains several files and directories which leads the deployment ( directly or indirectly ) a process between SCM and the destination servers. Capistrano handles this quite well, especially In case of any deployment error, there is this automated rollback capability which tries to keep all the target nodes in the cluster on the same version.
Read more

Type-Safe and Clean Coding: The Benefits of Type-Inference

Type-Safe and Clean Coding: The Benefits of Type-Inference

Type-interference is a great programming feature that helps coders write clean, readable code in a reasonable amount of time. Learn more here.

Writing type-safe language while maintaining less boilerplate code is an important aspect of programming languages in terms of developer’s productivity. Because type-safe code is less error-prone and less boilerplate code leads to more readable code, both together means reduced development time. Type-inference is a great programming language feature that maintains this balance.

Developers usually read more often than write. Thus, even if the source code will end up being processed by the computer, most of the time our focus is putting it into more human-readable form. At some point, we pay attention to the UX principles, like:

  • Humans have a limited attention span, so source code should help to spend this attention wisely. Information comes at a cost, so the longer the code is, the more overwhelming it is to read.
  • Sometimes, less means more. Short code may look brilliant, but it amplifies the time that others must spend on it. We should provide just enough information. Implicit values, if we do not abuse them, are the fix for this.

Read more