Questions tagged [distributed]
is for questions dealing with distributed systems, programs and programming.
33
questions
0
votes
0
answers
9
views
How to train huge ML model on multiple budget machines on web?
Is there a way to train huge models (1+ TiB of parameter data) being distributed on web? For example, I have a several hundreds of budget machines, each equipped with 2-4 GiB of RAM and with no ...
- 111
0
votes
1
answer
36
views
MQTT broker for fault-tolerance
Looking for a distributed MQTT broker that remains operational when one of its nodes fails for interdevice comunication using low-bandwith in time-sensitive scenarios. Its a low volume of messages and ...
1
vote
0
answers
7
views
is there some profiling and coverage library for rust?
This question is related to a common criticism of the model for sharing computing resources on blockchains (often seen on economics), based on accounting of participation of various users on the ...
3
votes
1
answer
32
views
Software with a database to lookup hashes of files
Sometimes it happens that an open source project adds files with a wrong license.
For example the C code of a game is GPL-2, but by accident a sound file from a proprietary source was copied to the ...
- 341
0
votes
0
answers
11
views
How to track requests and completion status in async systems?
I am trying to explore solutions to track requests/commands in Async systems. This is my use case:
Currently in our system we have 2 services communicating asynchronously with Service 1 sending an ...
- 101
0
votes
0
answers
21
views
A distributed file system with interactive conflict resolution?
I have searched high and low and cannot find anything suitable for my use case:
I’m looking for a solution that offers me to use the same files on both my mobile, laptop, desktop, home server and ...
- 101
1
vote
0
answers
22
views
Distributed FS for storing data in the single "master" node and duplicating that data, which is used on "worker" nodes
I am looking for a DFS for unified data access and caching used files within a small cluster.
We are currently using NFS to RAID node, and it seems not the best way to further scaling. So, I am ...
1
vote
0
answers
8
views
Framework/Tool for real-time inference with large input flow using Tensorflow
I'm working on a part of a project, that detects fraud from sensors' data using deep learning. Current steps are:
Receiving sensors' data (I'm currently receiving data from Kafka).
Predicting ...
- 11
0
votes
1
answer
42
views
looking for the distributed / parallel processing software?
I have 1 PC, 1 iMac and 5 laptops and use only 1 or 2 at a time. My hobby is developing web, video/audio editing. So when encoding videos, I would like to assign other laptops for parallel processing ...
- 101
2
votes
0
answers
80
views
Distributed arrays in Python
I am looking for a minimal interface to distributed ndarrays. I'd like to chunk my data in arbitrary blocks, scatter them across a cluster, run computations on the local numpy array, rechunk my data ...
- 21
1
vote
0
answers
128
views
Multi-leader or leaderless replication distributed DB (in-memory)
I am trying to find out few examples of multi-leader or leaderless distributed DB. It seems Redis is there but only enterprise version is having multi master support.
Any distributed DB which satisfy ...
- 111
1
vote
0
answers
28
views
Alternative analytics database for memsql and crate.io
Have been looking for real-time analytics database as a managed service that could scale horizontally and have column storage and support distributed joins. Data is not too big (around 2 billion rows ...
- 11
1
vote
2
answers
49
views
Distributed Database Systems
I am looking for an open source distributed SQL database system which will distribute the data across nodes but so that there is data redundancy (if one node fails, the data is still available without ...
- 11
7
votes
1
answer
926
views
Are there free tools/setups for cached, distributed C++ compiling with CMake on Windows?
This problem touches too many topics outside of the CMake, C++, Windows world I am familiar with. I kind of hit a wall of way to many options to explore each one. Actually I am hoping for someone ...
- 71
0
votes
0
answers
15
views
parallel processing
We asked our dear friends, there are several idle systems that we want to use with our network of other active systems, 4 weighing software applications, in parallel processing and maximum processing ...
1
vote
0
answers
198
views
Can someone suggest me a NoSQL database which is good for frequent updates?
I have heard that NoSQL databases like Cassandra, Couchbase etc are not good for frequent updates of data. They behave well when the data inserted doesn't change often. My use case is to save/update ...
- 11
0
votes
1
answer
36
views
Distributed data storage recommendations in mixed operating system environment for Python applications (Hadoop vs Databricks vs others)
I'm working on a big data project and have several old on-prem servers running a mix of operating systems (Ubuntu, CentOS, Windows 2012, Windows 10). One of the big reasons I would like a distributed ...
- 1
0
votes
0
answers
28
views
Publish/subscribe software to handle at least 10k requests/sec and 10k simultaneous connection
I am on the verge to start making a "Large distributed application", but i'm struggling to decide what software is the most adequate to do so.
My needs:
A publish/subscribe application, which ...
1
vote
0
answers
42
views
How to store 1.5 GB CSV files?
I am searching a solution to store files of 1.5 GB. There is a decade of file per day.
I am searching a solution to store those files.
I have already done some searches on HDFS and MongoDB. However, ...
- 111
0
votes
0
answers
22
views
Leader Election with Customization
I have multiple micro-service instances of same type. Each service instance will have connections to some of remote services. In below diagram, I have shown 3 service instance and 3 remote service. ...
- 101
2
votes
0
answers
23
views
Shareable List of Links With Notifications
So basically I am creating Tinycards decks for the vocabulary of each Duolingo Hebrew level that feature transliterations to practice learning not only the words and their meanings but also their ...
2
votes
0
answers
19
views
How can i run my service on a clustered environment?
How can i run my service on a clustered environment ?
Any products out there to help me out. This product should take care of
managing the cluster, cluster failover, rebalancing in case of node ...
1
vote
0
answers
31
views
Automated system testing and performance measurement for distributed system
I'm looking for a tool/combination of tools which would allow me to do system testing for application spanning multiple hosts.
What I'm looking for:
ability to spawn variable number of application ...
1
vote
0
answers
189
views
Distributed build system for Rust
I'm looking for a distributed build system for Rust that:
Is free, gratis and opensource
Is cross-platform and cross-compiles (from ARMv6, ARMv7, i386 and amd64 Linux and amd64 Windows for amd64 ...
- 1,236
4
votes
2
answers
198
views
A distributed (on-demand) build system
I'm tired of long build times …
I'm interested in a distributed build system, either at the lower-level of running the equivalent of make, and not knowing anything about anything other than ...
- 3,131
0
votes
0
answers
195
views
Alternative for HBase to store and query small files
I'm using Hadoop on my system to store my files (each file is very large: ~4GB). I also have small sqlite3 db files (~50MB) and I would like to store them (somewhere - I don't have shared storage) and ...
- 101
2
votes
0
answers
41
views
Distributed, indexed regular expression search
I'm looking for a distributed solution to do regular expression searches over a large directory structure with many nested directories and files. Ideally it will be also able to search file/directory ...
- 129
2
votes
0
answers
189
views
Cluster computing drop-in replacement for Python multiprocessing Pool
Python has a very handy multiprocessing built-in module. Particularly its Pool class allows simple parallelization. However, it only supports a shared-memory multiprocessing model, not a distributed ...
- 121
4
votes
1
answer
58
views
Distributed updates over LAN for Windows
Does software exist that would make it easy to distribute software updates for a variety of programs over LAN?
My job involves supporting numerous low-bandwidth offices with very expensive internet ...
- 61
2
votes
2
answers
341
views
Dense Linear Algebra C/C++ library for distributed computing
My goal is to invert a large dense matrix in a distributed environment.
I would like a library which would be "easily" integrated in a C++ application, thus it would be ideal that the library has a C/...
- 135
1
vote
2
answers
111
views
Version control system on distributed files or database?
Do you know of any "version control" or sync tool able to work transparently on files stored on distributed file systems (such as Hadoop) or using their own database to keep the files?
I'm looking ...
- 171
1
vote
0
answers
59
views
Distributed job queue with task locality?
Context
At present i have one server, which knows some tasks need to be scheduled at recurring intervals. So one process pushes such tasks into a message queue on a timer; and then other processes ...
- 21
5
votes
0
answers
210
views
Free distributed wiki software with full-text search, working even offline
I'm looking for a way to organize knowledge in my company.
Since we have to work disconnected from the company net many times, a server-client solution alone won't suffice.
The requirements are:
...
- 301