Full stack data processing
One of our clients is a world class leader in networking software and real-time content delivery all around the globe. This means they require processing millions of data records per second to be able to provide business value for their customers.
Aligned Research Group played a major role in transforming these data pipelines from batch processing to a real-time streaming. We wrote a number of analytic and ETL tools in different languages (Python, C++, Go, Scala) and provided both CI/CD (automated testing and DevOps) and maintenance for them. Most of this infrastructure is managed by Kubernetes + Ansible and uses Apache Kafka as a main data bus.
This real-time data workflow involves multiple moving components, including anomaly detection, in-memory graph database and a set of GPU-powered algorithms, like neural network for correlation analysis and Locky malware detection.