Sandbox DB: универсальная песочница для погружения в Big Data, аналитику и визуализацию
Запускайте PostgreSQL, ClickHouse, Airflow, Superset и другие инструменты одним кликом: учите, экспериментируйте, осваивайте новое!
Sandbox DB: универсальная песочница для погружения в Big Data, аналитику и визуализацию
Запускайте PostgreSQL, ClickHouse, Airflow, Superset и другие инструменты одним кликом: учите, экспериментируйте, осваивайте новое!
New post about how to write data from a Apache Spark DataFrame into a Elasticsearch/Opensearch database #datascience #databricks #elasticsearch #opensearch #bigdata #apachespark #spark #tech #programming #python:
https://pedro-faria.netlify.app/posts/2025/2025-03-16-spark-elasticsearch/en/
New call for proposals: #Spark enables researchers from all disciplines to test or develop novel and unconventional scientific approaches, methods, theories or ideas within a short time.
Submission deadline: 4 March 2025.
https://www.snf.ch/en/CVNR0Q5f3P32Cg9f/news/spark-call-for-proposals
Just caught up with the recent Delta Lake webinar,
> Revolutionizing Delta Lake workflows on AWS Lambda with Polars, DuckDB, Daft & Rust
Some interesting hints there regarding lightweight processing of big-ish data. Easy to relate to any other framework instead of Lambda, e.g. #ApacheAirflow tasks
This is a customer-facing role, so if that's not your thing, keep scrolling.
TLDR: If you know Hadoop and live close enough to Belfast to commute, you should apply.
I've posted this before, but it's been a little while #fedihire. Also, adding some additional information this time. This is my team. We are already on three continents and 6 timezones, but #Belfast is a new location for the team. I know literally nothing about the office.
I know a lot of places Hadoop is the past, and sure we see a ton of #Spark (I do not understand why that is not listed in the job description but maybe because they want to emphasis that we need hadoop expertise?). You can see all the projects we support at https://www.openlogic.com/supported-technology
It depends on how you count, as I was on two teams during tradition, but I've been on this team for over 5 years now. It's a great team. I've been with the company now right at 7 years. I cannot say how we compare to Belfast employers but this is well more than double where I have stayed at any other employer (even if you count UNC-CH as a single employer rather than the different departments, I've beat them by well over a year at this point).
My manager has been on this team for almost 15 years. His manager has been with this team for almost as long as me, but with the company much longer. His manager has been here almost as long as me (I actually did orientation with him). His manager is a her and she's been here almost as long as me. So, obviously, this is a place where people want to stay!
Our team has a lot of testosterone, but when I started, our CEO was a woman. The GM for the division is a woman.
My manager is black. The manager of our sister team is black.
I think you'll find our team and company is concerned about your work product and not how you dress, what bathroom you use, or the color of your skin.
If you take a look at our careers page, you'll see this:
Work Should Be Fun
There’s always something to look forward to as a Perforce employee: scavenger hunts, community lunches, summer events, virtual games, and year-end celebrations just to name a few.
We take that shit seriously. Nauseatingly so sometimes, lol.
Actually, we take everything on the careers page seriously, but I know from experience that some places treat support like they are a shoe sole to be worn down. Not so here. It's not all rainbows and sunshine, of course. The whole point is that the customer is having an issue! Our customers treat us with respect because management demands that they do.
------
The Director of Product Development at Perforce is searching for a Enterprise Architect (#BigData Solutions) to join the team. We are looking for an individual who loves data solutions, views technology as a lifestyle, and has a passion for open source software. In this position, you’ll get hands on experience building, configuring, deploying, and troubleshooting our big data solutions, and you’ll contribute to our most strategic product offerings.
At OpenLogic we do #opensource right, and our people make it happen. We provide the technical expertise required for maintaining healthy implementations of hundreds of integrated open source software packages. If your skills meet any of the specs below, now is the time to apply to be a part of our passionate team.
Responsibilities:
Troubleshoot and conduct root cause analysis on enterprise scale big data systems operated by third-party clients. Assisting them in resolving complex issues in mission critical environments.
Install, configure, validate, and monitor a bundle of open source packages that deliver a cohesive world class big data solution.
Evaluate existing Big Data systems operated by third-party clients and identify areas for improvement.
Administer automation for provisioning and updating our big data distribution.
Requirements:
Demonstrable proficiency in #Linux command-line essentials
Strong #SQL and #NoSQL background required
Demonstrable experience designing or testing disaster recovery plans, including backup and recovery
Must have a firm understanding of the #Hadoop ecosystem, including the various open source packages that contribute to a broader solution, as well as an appreciation for the turmoil and turf wars among vendors in the space
Must understand the unique use cases and requirements for platform specific deployments, including on-premises vs cloud vs hybrid, as well as bare metal vs virtualization
Demonstrable experience in one or more cloud-based technologies (AWS or Azure preferred)
Experience with #virtualization and #containerization at scale
Experience creating architectural blueprints and best practices for Hadoop implementations
Some programming experience required
#Database administration experience very desirable
Experience working in enterprise/carrier production environments
Understanding of #DevOps and automation concepts
#Ansible playbook development very desirable
Experience with #Git-based version control
Be flexible and willing to support occasional after-hours and weekend work
Experience working with a geographically dispersed virtual team
https://jobs.lever.co/perforce/479dfdd6-6e76-4651-9ddb-c4b652ab7b74
Day 4 of 12: Understanding key terms for data professionals
As more and more data is generated, we need technologies to process it efficiently. Companies also want to be able to process data in (near) real time. This is where tools such as Spark or Kafka (Big Data Technologies) come into play.
Today's Small Practical Project:
Develop a small pipeline with Python that simulates, processes and saves real-time data: For example, simulate real-time data streams of temperature values. Then check whether the temperature exceeds a critical threshold value. As an extension, you can plot the temperature data in real time.
We’re thrilled to announce the release of orbital 0.3.0!
The orbital package allows you to run predictions from tidymodels workflows directly inside databases. This new version brings support for classification models and the `augment()` function.
Read more in the tidyverse blog: https://www.tidyverse.org/blog/2025/01/orbital-0-3-0/
#Spark niestety w moim flow upadł. Wersja na desktopa jest dla mnie nieużywana z racji tego, że korzystam z wielu funkcji macOS (Tagi, Skróty klawiszowe: Sent in Mail w menu, pracuje na katalogach i regułach na serwerach). Mail śmiga i się integruje, w Sparku muszę albo kombinować jak coś obejść, albo się nie da (np. z podglądu wydruku często korzystam z opcji Save to PDF i Sent in Mail i tej drugiej nie udało mi się zastąpić).
Na mobilnym urządzeniu jest lepiej do mojego flow, ale wartości dodane czyli inteligentne katalogowanie nie jest mi aż tak potrzebne jak mam reguły.
Tak, że ten. Zachwyt aplikacja ok, jest super. Przydatność - to zależy - u mnie nie daje korzyści.
#FirstCape Group Dramatically Boosts Stake in #Spark #NewZealand to Over 5%, Fueling #DataCenter Growth Amid Spark’s Major Restructuring
#Spark New Zealand Sells Final Stake in #Connexa for $181 Million, Strengthens #DataCenter Strategy Amid Major Shift
AI-Powered @github Spark lets you build apps using natural language
https://www.admin-magazine.com/News/AI-Powered-GitHub-Spark-Released-for-Creating-Micro-Apps
#GitHub #Spark #AI #OpenSource #NaturalLanguage #apps #FOSS #ArtificialIntelligence
GitHub Copilot підтримуватиме ШІ-моделі від Anthropic, Google й OpenAI https://itc.ua/ua/novini/github-copilot-pidtrymuvatyme-shi-modeli-vid-anthropic-google-j-openai/ #GitHubUniverse2024 #GitHubCopilot #Технології #Новини #GitHub #OpenAI #Spark #Софт
[17:21] ‘What a mad few months’ – Irish rap hit The Spark longlisted for 2025 Grammys
The Kabin and Lisdoovarna Crew's viral hit The Spark has been longlisted for the 2025 Grammy Awards.
https://www.independent.ie/irish-news/what-a-mad-few-months-irish-rap-hit-the-spark-longlisted-for-2025-grammys/a1016205444.html
#Kabin #LisdoovarnaCrew's #Spark #2025 #GrammyAwards
If you’re an Apache #Spark user, you benefit from its speed and scalability for #BigData processing.
However, you might still want to leverage #RStats’s extensive ecosystem of packages and intuitive syntax. One effective way to do this is by writing user-defined functions (UDFs) with sparklyr.
UDFs enable you to execute R functions within Spark, harnessing Spark’s processing power and combining the strengths of both tools.
Learn more in a recent blog post → https://posit.co/blog/databricks-udfs/
Are you a Spark user who prefers writing in R? User-defined functions with sparklyr might be what you need
With `spark_apply()`, you can write functions in #RStats and use them in #Spark queries.
Learn more in the blog post: https://posit.co/blog/databricks-udfs/
Ein 50 Jahre alter Verstärker von Fender oder Marshall oder Hiwatt ist heute ein begehrter Klassiker. Und er funktioniert (wenn gewartet) heute immer noch, wie vor 50 Jahren: Gitarre, Kabel, Strom, fertig.
Was wird aus den Amps, die heute mit #Bluetooth, #DSP und App auf den Markt kommen: wird es da in 5 Jahren noch eine angepasste und aktualisierte App für geben? Oder werden die Funktionen langsam sterben, wenn die passenden Handys verschwinden?
Here's a link to CBC Radio One Spark, and its final episode, about reasons for hope in the tech community.
The headliner was us all right here, members of the #Fediverse.
The sparklyr package and friends have been getting some important updates in the past few months!
sparklyr is a package that allows you to interact with Spark using familiar R interfaces, such as dplyr, broom, and DBI. You can also gain access to Spark's distributed Machine Learning libraries, Structure Streaming, and ML Pipelines from R.
Read more in the blog post: https://blogs.rstudio.com/ai/posts/2024-04-22-sparklyr-updates/
My company is hosting a webinar in 5 minutes! buff.ly/3VWGQi9
Come see a live demo on 100TB of data and talk to our engineers. (read not a marketing presentation) #dataanalytics #gpu #hpc #etlongpu #spark #datascience #data