Introducing Vector, a high-performance data router, written in Rust

Yesterday, the team at Timber.io, a cloud-based logging platform, released Vector, a high-performance observability data router that makes transforming, collecting, and sending logs, metrics, and events easy.

One of the reasons behind building Vector was to integrate mtail's functionality into a bigger project. mtail is a tool which is used for extracting metrics from application logs. Licensed under the Apache License, Version 2.0, Vector decouples data collection and routing from user services that give users the control and data ownership. Vector which written in Rust, compiles to a single static binary and it has been designed to be deployed across the entire infrastructure. 

Concepts of Vector

Following is a diagram depicting the basic concepts that Vector comprises of:

Image source: Vector

Sources

When Vector ingests data it proceeds to normalize that data into a record, which sets the stage for easy and consistent processing of the data. Examples of sources include syslog, tcp, file, and stdin.

Transforms

Transform modifies an event or the stream as a whole like a filter,  parser, sampler, or aggregator. 

Sinks

A sink is a destination for events and its design and transmission method is controlled by the downstream service it is interacting with. For instance, the TCP sink will stream individual records, while the S3 sink will buffer and flush data.

Features of Vector

Memory efficient and fast

Vector is fast and memory-efficient and doesn't have a runtime and garbage collector.

Test cases

Vector involves performance and correctness tests, where the performance tests measure performance and capture detailed performance data, whereas, correctness tests verify behavior. 

The team behind Vector has also invested in a robust test harness that provides a data-driven testing environment. 

Here are the test results:

Image source: GitHub

Processing data

Vector is used for collecting data from various sources in various shapes. It also sets the stage for easy and consistent processing of the data.

Serves as a single tool

It serves as a light-weight agent as well as a service that works as a single tool for users.

Guarantee support matrix

It features a guarantee support matrix that helps users understand their tradeoffs.

Easy deployment

Vector cross-compiles to a single static binary without any runtime.

Users seem to be happy about this news as they think Vector is useful for them. A user commented on HackerNews, "I'm learning Rust and eventually plan to build such a solution but I think a lot of this project can be repurposed for what I asked much faster than building a new one. Cheers on this open source project. I will contribute whatever I can. Thanks!!"

It seems more metrics-focused sources and Sinks are expected in Vector in the future. A member from the Vector project commented, "It's still slightly rough around the edges, but Vector can actually ingest metrics today in addition to deriving metrics from log events. We have a source component that speaks the statsd protocol which can then feed into our prometheus sink. We're planning to add more metrics-focused sources and sinks in the future (e.g. graphite, datadog, etc), so check back soon!"

To know more about this news, check out Vector's page.

Implementing routing with React Router and GraphQL [Tutorial]
TP-Link kept thousands of vulnerable routers at risk of remote hijack, failed to alert customers
Amazon buys ‘Eero’ mesh router startup, adding fuel to its in-house Alexa smart home ecosystem ambitions