Senior Data Engineer
Metadata Inc
Senior Data Engineer (Remote)
We are a team of highly dedicated professionals who are making a breakthrough in B2B marketing by introducing an autonomous technology powered by AI to replace the daily mundane tasks of marketing operations teams, empowering them to execute 100x faster and more efficiently, freeing up time for creative and strategic tasks.
The company is REMOTE with members spread out across 14 countries and 3 continents. We move fast but scientifically; we are agile but disciplined. We value and reward dedication, hard work, and a can-do attitude. We embrace equality, diversity, and authenticity. “Givers” will thrive in our team!
We are seeking a Senior Data Engineer who has a strong desire and the drive to contribute to the next generation of our innovative product. As a SDE at Metadata, you will have a chance to contribute on a globally distributed team from 10 different countries, make deep technical contributions and decisions, manage your own priorities without supervision, participate in innovative tech design, understand how your work fits in with related projects or components, and help the team course-correct when necessary.
It is a rare opportunity to boost your career growth, learn a lot, meet talented people, and work with cutting-edge technologies to transform B2B Marketing.
The learning curve is steep and challenging, and the day-to-day life of aIf this sounds like the right opportunity for you, we want to hear from you as soon as possible! Senior Data Engineer requires sharp focus, exceptional attention to detail, an analytical approach, and continuous learning.
If this sounds like the right opportunity for you, we want to hear from you as soon as possible!
Responsibilities
● Design and implement workflows for ingestion and transformation of new data sources that will be used in the company.
● Report generation and distribution based on the existent data and new external sources of data ingested.
● Evolve the current Data Engineering infrastructure to make it scalable and get it ready for the predicted future volume
● Work on maintenance and redesign of the current data lake to empower analysts and ML team.
● Maintain and deploy ML models, set up the necessary infrastructure to run those models, and distribute the predictions to the appropriate place.
REQUIREMENTS
Required Skills
● 7+ years of experience as a Software Engineer
● Proficient in Python and SQL.
● Experience with Apache Spark, Python libraries for parallelization and creation of rest API clients, and data manipulation like Pandas.
● Knowledge on MySQL, and relational databases.
● Database normalization, denormalization, star and snowflake schemas.
● Experience with search engines such as ElasticSearch, Solr
● Experience with key/value databases like Redis.
● Experience with Streams and Message Brokers such as RabbitMQ and Kafka.
● Good knowledge of data formats like Json, Parquet, Delta, Avro, Protocol Buffers...
● Should be comfortable with Linux (basic scripting language), Gitlab, Jira and a good grasp of the concept of DevOps and CI/CD
● Proven experience with taking a feature request, providing tech design, implementing end-to-end, performing code review
● Familiar cloud technologies such as Azure, AWS or DigitalOcean
● Excellent understanding of design patterns, data structures, and algorithms
● Proficient logical and analytical skills
● Familiar with working in Agile environment and remote teams
Similar Jobs
