dhilipsiva - Musings

On building things that outlive their meaning

Unknown — Thu, 28 May 2026 00:00:00 +0000

The universe does not owe your software a reason to exist. It does not owe your cluster a quorum, your proof a model, or your name a legacy. This is usually delivered as bad news. I have come to read it as the most freeing license a builder can be handed.</p>

If nothing is owed, then nothing is wasted either. You are free to build the thing precisely because it does not have</em> to matter. The meaning is not discovered in the artifact; it is authored</strong> into it, by you, at the moment you decide to begin.</p>

A quine is a program that, run, prints its own source. It needs no input from the world to justify its output. It is its own reason. — from the margins of an old notebook</cite></p> </blockquote>
The fixed point as a way of life</h2>
In mathematics, a fixed point</strong> of a function f</code> is a value x</code> where f(x) = x</code> — apply the function and nothing moves. Consensus is a fixed point. A quine is a fixed point. A self-consistent worldview is a fixed point. The trick of building systems that last is to find the place where the system stops needing the world to hold it up, and instead holds itself.</p>
Here is the smallest honest version of the idea, in Rust:</p>
// the network's opinion of itself, iterated to a fixed point</span> fn</span> settle</span><</span>T</span>:</span> Eq</span> + Clone</span>>(</span>mut</span> x:</span> T</span>,</span> f:</span> impl</span> Fn</span>(&</span>T</span>)</span> -> T</span>)</span> -> T</span> {</span> loop</span> {</span> let</span> next = f</span>(&</span>x);</span> if</span> next == x {</span> return</span> x; }</span> // ∴ eval(x) = x</span> x = next;</span> }</span> }</span></code></pre> You can call settle</code> on anything that knows how to compare itself to a previous version of itself. Nodes in nibli</code>'s federated gossip do exactly this: they exchange knowledge until no opinion changes, and then they call that agreement. The optimism is not that agreement is guaranteed — it isn't. The optimism is that it is worth iterating toward anyway</em>.</p> Three properties of things that last</h3> When I look at the systems that have outlived their original purpose, they tend to share a few traits:</p> They are deterministic.</strong> Same inputs, same history, every time. A future maintainer can replay the past instead of guessing at it.</li> They are small.</strong> A 12kB core can be understood, ported, and resurrected. A 12MB one becomes a fossil the moment its authors leave.</li> They are honest about limits.</strong> They say what they cannot do, in writing, near the code that cannot do it.</li> </ul> An aside on undecidability</h4> There are questions our systems provably cannot answer — whether an arbitrary program halts, chief among them. This is not a flaw to engineer around so much as a horizon to build up to.1</a> Knowing the horizon exists is what lets you stop pretending you can see past it.</p> Writing across two scripts</h2> I think in English and in Tamil, and the two scripts disagree about how a sentence should breathe. Long-form type has to hold both without one looking like an afterthought. Here is the same idea in both — the line-height has room for Tamil's tall vowel signs:</p> ஒரு நிரல் தன்னைத்தானே எழுதிக்கொள்ளும்போது, அதற்கு வெளியே இருந்து எந்த அர்த்தமும் தேவையில்லை. அதுவே அதன் காரணம். (A program that writes itself needs no meaning from outside; it is its own reason.)</span></p> Naming things well is the same act in either language: you are choosing the fixed point a future reader will resolve the symbol to. Choose generously.</p> நம்பிக்கையோடு கட்டு. ஒன்றும் கடன் இல்லை. — build with hope; nothing is owed</cite></p> </blockquote> So: build anyway</h2> The nihilism is the premise, not the conclusion. The conclusion is a working system, committed to disk, that did not have to exist and does. That is the whole optimistic move — to look at an indifferent universe and add one more fixed point to it, on purpose.</p> One must imagine the compiler happy.</p> The halting problem: there is no general algorithm that decides, for every program-input pair, whether the program eventually halts. Turing, 1936. ↩</a></li></ol></div>

How I Annoyed My Way to Clash Royale's Legendary Arena Unknown — Wed, 11 Jun 2025 00:00:00 +0000 After months of grinding, defending, and annoying a lot of opponents, I've finally done it. I've reached the Legendary Arena at 9000 trophies. And I did it not with Golem or Hog Rider, but with my own weird, super-defensive deck that many people would say is crazy.</p> Today, I want to celebrate by sharing my full strategy with you all. No secrets held back. If you love to play smart, control the game, and drive your opponent mad with patience, then this post is for you.</p> My Tools of Trade: The Deck</h2> First things first, here's the deck list. All my cards are at Elite Level, which makes a big difference.</p> Buildings:</strong> Cannon, Tesla Tower, Mortar — The twin pillars of my defense.</li> Troops:</strong> Valkyrie, Knight, Dart Goblin, Ice Wizard — My ground control crew.</li> Special Card:</strong> Suspicious Bush — The sneaky little troublemaker.</li> Princess Tower Support:</strong> Cannoneer — For that extra punch against heavy troops.</li> </ul> My Main Funda: The "Susanoo" Defense</h2> My main playstyle is pure defense. I like to think of it as activating "Susanoo" from Naruto — a perfect, almost unbreakable defense.</p> From the start of the game, I place my buildings right in the middle. This central fortress pulls any building-targeting troop (like Hog Rider or Golem) into a kill zone. A great trick against Hog Riders or Balloons is to place a building in a way that drags them to the middle, ensuring your King Tower gets activated early</strong>. An active King Tower is a massive defensive advantage for the rest of the game!</p> The Secret Sauce: Play Mind Games, Not Just Cards</h2> This deck is all about psychological warfare. My goal is to annoy the opponent so much that they start making mistakes.</p> Forcing Bad Choices:</strong> If the enemy has a Fireball, I deliberately place my buildings and troops separately. They can't get value by hitting both my tower and my building. They have to choose, and it frustrates them like anything.</li> Conditioning Your Opponent:</strong> This is a top-level secret. For the first few minutes, I use the same defensive patterns. The opponent gets comfortable. They think, "Okay, I know what this guy is going to do." They become predictable. And then, at a crucial moment, I break the pattern — a slightly different placement, an unexpected push — and it completely throws them off their game.</li> </ul> The Suspicious Bush</strong> is my ultimate mind-game tool. I drop it at the back. It moves slowly, it's invisible to troops, and the opponent has</em> to deal with it. It's a brilliant way to force them to spend elixir and distract them while I prepare my real plan.</p> The "Aha!" Moment: The Surprise Attack</h2> After defending and defending, I'm constantly tracking the opponent's cards and elixir. There are moments in a game, usually in 2x elixir, where I know for a fact that the opponent is low on elixir and their main defensive card is out of their hand.</p> BAM!</strong></p> I immediately drop my Mortar right at the bridge</strong> or in the center. After seeing my mortar being used for defense throughout the game, I surprise them with a offensive placement when they are at their weakest. They panic. They drop cards frantically near the Mortar to kill it.</p> But it's too late. I've already anticipated their move and dropped a Tesla or a Knight to protect my Mortar. Once the panic starts, I push it further by spamming whatever I have at the bridge to keep them overwhelmed.</p> The Real Magic: Skills You Need to Master</h2> 1. Become a Master Counter (Elixir & Cards):</strong> Don't just guess when the opponent is low. Know it. Watch every card they play and do the maths. More importantly, track their 4-card cycle. The best time to attack isn't just when they are low on elixir, but when their best counter to your Mortar is not in their hand.</li> 2. Master Your Placements:</strong> Learn the exact tile to place your buildings to pull every type of troop and to activate your King Tower. A one-tile difference is the difference between winning and losing.</li> 3. Control Your Card Cycle:</strong> This is key. When the pressure is low, quickly play your cheap cards like Dart Goblin or Suspicious Bush (even at the back). This ensures your most important defensive cards, like Tesla or Valkyrie, are always in your hand right when you need them most.</li> 4. Know When to Switch (Defense to Attack):</strong> You are a defender 90% of the time. But you need to have the guts to switch to a full-on attacker in that 10% window of opportunity. Be patient, then be aggressive, then immediately be patient again if your push fails.</li> </ul> The One Big Weakness (And How to Fix It)</h2> I'll be honest, the deck struggles against heavy air decks like Lava Hound with Balloon. It's tough. If you are facing a lot of air decks, here's the one change I sometimes make:</p> Swap Cannon for Fireball or Poison.</strong></p> Losing a building hurts my core defense, but having a big spell lets me damage support troops like Wizards hiding behind a tank, and it can help me chip down a tower to win the game.</p> Final Thoughts</h2> Reaching Legendary Arena wasn't about raw power. It was about patient defense, clever psychological play, meticulous elixir management, and strategic surprise. This deck is my proof.</p> Try this approach yourself, be patient, and watch your opponents crumble under the pressure of a true defensive master!</p> Black Holes, True Vacuum, and Cosmic Fabric: An Enthusiast's Ramblings Unknown — Tue, 20 Aug 2024 00:00:00 +0000 Disclaimer: I am an Idiot</strong></h2> First things first, let me make something absolutely clear — I am not a physicist. In fact, I am very far from one. I'm just an idiot who is extremely enthusiastic about science in general and a fanboy of theoretical physics in particular. I've read a bit, watched a lot of YouTube videos, and done my share of daydreaming about the mysteries of the universe. So, please take everything I say with a grain of salt because I mostly don't have a clue about what I'm talking about. This blog post is more of a thought experiment (not to mention factually wrong), a wild idea that popped into my head, and I'm putting it out here in the hope that maybe someone would just indulge my mediocrity.</p> What if We Were Wrong About Vacuum Decay Expanding at the Speed of Light?</strong></p> Before diving into my hypothesis (Well, not really a hypothesis — just an imagination of mine), let's entertain the possibility that the widely accepted theory about vacuum decay expanding at nearly the speed of light might be incorrect (I know. I know it is ridiculous, we might as well assume that one could fit an entire, fully-grown elephant inside a tennis ball). What if, instead, the expansion happened at a slower or different speed, altering our entire understanding of how this catastrophic event would unfold? This seemingly small change could have profound implications for our theories about the universe, particularly when we consider phenomena like black holes.</p> The Hypothesis: Black Holes as Regions of True(er) Vacuum</h1> So, here's the idea I've been toying with: What if black holes are regions of space-time where the quantum fields have undergone a phase transition to a true (or truer) vacuum? Essentially, this means that black holes could be areas where vacuum decay, something that theoretical physicists speculate about, has already happened.</p> If vacuum decay didn't necessarily expand at almost the speed of light, then perhaps the extreme conditions surrounding a collapsing star could trigger this phase transition locally, leading to the formation of a black hole as we observe it today. This would mean that what we think of as a black hole could actually be a region of space where a more stable vacuum has replaced the original false vacuum.</p> The Role of a Collapsing Star</h1> Let's add another layer to this daydream. Imagine a massive star reaching the end of its life. As it collapses under its own gravity, the conditions become so extreme — high energy densities, intense gravitational forces — that this collapse could trigger a vacuum decay. In this scenario, the collapse of the star doesn't just create a black hole as we traditionally understand it; it catalyzes a phase transition, turning the region inside the event horizon into a true vacuum or something closer to it. This new vacuum state, being more stable or energetically favorable, would replace the old one, and the result would be what we observe as a black hole.</p> Event Horizon as a Fabric of Cosmic Strings</h1> Now, extending this idea further: What if the event horizon — the boundary of a black hole beyond which nothing can escape — isn't just a simple boundary? What if it's actually a "fabric of cosmic strings," a sort of mesh made up of these one-dimensional defects in space-time that separates different vacuum states inside and outside the event horizon? These cosmic strings could represent a gradient between regions of false vacuum outside the event horizon and regions that have undergone or are undergoing a phase transition to a truer vacuum inside.</p> Hawking radiation, the process by which black holes are thought to slowly lose mass, could be a byproduct of the interaction between these vacuum states across this fabric of cosmic strings. The radiation we observe might be the result of the quantum states outside the fabric gradually aligning with those inside it.</p> Potential Resolutions: What Could This Hypothesis Explain?</h1> Let's start with what this hypothesis might resolve or explain:</p> Event Horizon as a Dynamic Interface:</strong> The idea of the event horizon as a fabric of cosmic strings separating different quantum states could give us a more intuitive understanding of why this boundary exists and how it operates. Instead of just being a point of no return, it could be an active interface between two distinct regions of space-time.</p> Hawking Radiation as a Quantum Process:</strong> If Hawking radiation is the result of quantum states aligning across the fabric of cosmic strings, it might offer a deeper explanation for why this radiation occurs in the first place. This could potentially tie in more closely with the fluctuations in quantum fields that are known to happen near the event horizon.</p> Black Hole Thermodynamics:</strong> The relationship between a black hole's entropy and the area of its event horizon is well understood in the current framework of black hole thermodynamics. This hypothesis might offer a new perspective by suggesting that the entropy is related not just to the event horizon area, but also to the interaction between different vacuum states across this fabric. The fabric of cosmic strings could be seen as a mediator of thermodynamic processes, linking the internal and external states of the black hole in a more fundamental way.</p> No-Hair Theorem:</strong> The no-hair theorem states that black holes can be completely described by just three properties: mass, charge, and angular momentum. They do not retain any other information about the matter that formed them. If the event horizon is a fabric of cosmic strings, it might suggest that additional information (related to the nature of the vacuum states) could be encoded in this boundary. This could potentially refine our understanding of black hole properties, allowing for a more nuanced view that still respects the theorem but adds depth to what those "hairs" might represent in terms of quantum states.</p> Potential Conflicts: Where This Hypothesis Might Fall Apart</h1> Now, let's tackle the elephant in the room — the potential problems that this hypothesis might run into:</p> Spacetime Geometry and General Relativity:</strong></p> Conflict:</strong> This hypothesis might conflict with the established geometry of space-time around black holes, as predicted by general relativity. The current solutions to Einstein's field equations, like the Schwarzschild or Kerr metrics, work well in explaining what we observe. Any new model would need to either fit within these existing solutions or replace them entirely.</li> Potential Reconciliation:</strong> If the fabric of cosmic strings is a more detailed description of what we currently understand as the event horizon, it might be possible to reconcile this idea with general relativity. Perhaps the fabric is a manifestation of the extreme warping of space-time predicted by Einstein's equations, with the vacuum state transition being a quantum phenomenon that occurs within this warped space-time.</li> </ul> Black Hole Thermodynamics:</strong></p> Conflict:</strong> The relationship between a black hole's entropy and the area of its event horizon is deeply entrenched in our current understanding of black hole thermodynamics. If we introduce the idea of a fabric of cosmic strings, we must ensure that this new interpretation aligns with the well-established principles of thermodynamics. We would need to explore whether the fabric introduces new variables or modifies the existing entropy-area relationship in a consistent manner.</li> Potential Reconciliation:</strong> The fabric of cosmic strings could be seen as an additional layer that complements our understanding of black hole entropy. It might provide a more fundamental explanation for why entropy scales with the event horizon area, potentially tying it to quantum field interactions across different vacuum states. If this can be mathematically formalized, it could enhance our understanding rather than contradict it.</li> </ul> Observational Evidence:</strong></p> Conflict:</strong> The hypothesis would also need to account for all the observations we have of black holes, such as gravitational waves from black hole mergers, the shadow of M87* captured by the Event Horizon Telescope, and the behavior of light and matter near black holes. These phenomena are well explained by our current understanding, so any new hypothesis must be able to explain these too.</li> Potential Reconciliation:</strong> If the fabric hypothesis can predict similar observational outcomes to current theories, it might not conflict with existing evidence. For instance, if the fabric's properties lead to the same gravitational effects as predicted by general relativity, we could still explain gravitational waves and the behavior of light. The key would be ensuring that the fabric's influence is consistent with what we observe.</li> </ul> No-Hair Theorem:</strong></p> Conflict:</strong> The simplicity of the no-hair theorem is one of its strengths. If we add the idea of a fabric with additional quantum information, we risk complicating the theorem. Any additional characteristics or information encoded in the event horizon must be consistent with the observation that black holes appear remarkably simple and must not lead to observable contradictions, such as unexpected radiation or particles that haven't been observed.</li> Potential Reconciliation:</strong> The fabric of cosmic strings might encode information in a way that doesn't violate the no-hair theorem but rather adds a layer of depth to it. For example, this additional information could be inherently inaccessible to outside observers, meaning it doesn't manifest in observable properties that would contradict the theorem. It could be analogous to quantum entanglement, where information exists but isn't directly observable in a way that violates established principles.</li> </ul> Cosmological Implications:</strong></p> Conflict:</strong> If black holes are regions where a vacuum decay has already occurred, why don't we see similar effects elsewhere in the universe? The hypothesis would need to explain why this phase transition is confined to black holes and how it impacts the broader cosmos. Moreover, if vacuum decay can be triggered by collapsing stars, this might have broader implications for our understanding of the universe's stability.</li> Potential Reconciliation:</strong> It's possible that the conditions required to trigger this vacuum transition are so extreme that they only occur in the cores of collapsing stars or other similarly intense environments. This would confine vacuum decay to black holes and prevent it from spreading across</li> </ul> Why Am I Writing This?</strong></p> Honestly, I'm writing this because I hope someone, just as lunatic as I am, out there might find it interesting to talk about fringe ideas like this. Maybe they'll laugh it off, or maybe they'll take it as an opportunity to dive into some interesting physics and give me a clearer understanding of how black holes and vacuum decay actually work.</p> Also, I can't take full credit for putting these ideas together. I had a lot of help from ChatGPT, my trusty AI assistant, who helped me organize my thoughts and flesh out this post.</p> So, there it is — a wild idea from a science enthusiast who is more clueless than not.</p> Harnessing Solar Power for Sustainable Water Desalination Unknown — Mon, 06 May 2024 00:00:00 +0000 Hello, everyone!</p> I'm exploring an innovative approach to convert saltwater into drinkable freshwater using solar energy, and I'm eager to gather insights and feedback from experts and enthusiasts alike.</p> The Idea</h1> The concept is simple yet potentially transformative. It involves using a massive magnifying lens to concentrate solar energy onto a large container of seawater. The goal is to boil the water at its center, reducing salt deposition on the container walls and instead allowing it to settle at the bottom as sediment. This sediment can then be periodically removed, ensuring efficient operation.</p> Why This Approach?</h1> Traditional methods of desalination, like boiling or reverse osmosis, require a lot of energy and can be expensive. They also tend to leave behind salt deposits that can degrade equipment over time. By using solar power, we aim to minimize energy costs and reduce the environmental footprint of desalination.</p> Location and Setup</h1> I believe the best location for this setup would be near coastal areas where sunlight is abundant throughout the year. The use of a high-grade optical lens to focus sunlight precisely could potentially bring water to a boil without the need for external energy sources.</p> Challenges Ahead</h1> Convection Currents</strong>: Heating water in the center might lead to convection currents that could distribute salts throughout the container. We need to think about how to manage this to prevent salt from depositing on the container walls.</li> Materials</strong>: The choice of materials for the container and the lens is crucial. They need to withstand high temperatures and corrosive saltwater. Options like stainless steel or titanium for the container and high-grade optical glass for the lens are on the table.</li> Scaling</strong>: How do we scale this setup to make it practical for large-scale water production? Modular setups might be the answer, allowing for easier maintenance and adaptability.</li> </ol> Economic and Environmental Considerations</h1> The initial costs, including the lens, structural setup, and land, could be significant. However, the use of local resources and materials could help in reducing these expenses. Environmentally, using solar power reduces carbon emissions dramatically, but we must also consider the lifecycle of the materials used.</p> Seeking Your Thoughts</h1> I'm looking for feedback on:</p> Potential design and material suggestions.</li> Insights on solar energy concentration and heat management.</li> Ideas for scaling and possibly automating the process.</li> </ul> Your expert opinions, thoughts, and critiques would be incredibly valuable as I refine this idea further. Let's discuss the potential of solar-powered desalination and how we can overcome the associated challenges together!</p> Thank you for reading, and I look forward to your insights!</p> Analyzing Python Compression Libraries: zlib, LZ4, Brotli, and Zstandard Unknown — Mon, 29 Apr 2024 00:00:00 +0000 GitHub: https://github.com/dhilipsiva/py-compress-compare</a></p> When dealing with large volumes of data, compression can be a critical factor in enhancing performance, reducing storage costs, and speeding up network transfers. In this blog post, we will dive into a comparison of four popular Python compression libraries—zlib, LZ4, Brotli, and Zstandard—using a real-world dataset to evaluate their performance in terms of compression ratio and time efficiency.</p> The Experiment Setup</h2> Our test involved a dataset roughly 581 KB in size, named sample_data.json. We executed compression and decompression using each library as follows:</p> Compression was performed 1000 times.</li> Decompression was repeated 10,000 times.</li> </ul> This rigorous testing framework ensures that we obtain a solid understanding of each library's performance under heavy load.</p> Compression Ratio</h2> The compression ratio is a key metric that represents how effectively a compression algorithm can reduce the size of the input data. Here's how each library scored:</p> Zlib achieved a compression ratio of 27.84,</li> LZ4 came in at 18.23,</li> Brotli impressed with a ratio of 64.78,</li> Zstandard offered a ratio of 43.42.</li> </ul> From these results, Brotli leads with the highest compression ratio, indicating its superior efficiency in data size reduction. Zstandard also shows strong performance, while LZ4, though lower, still provides a reasonable reduction.</p> Compression Time</h2> Efficiency isn't just about space savings; time is equally crucial. Here's how long each library took to compress the data:</p> Zlib: 7.34 seconds,</li> LZ4: 0.13 seconds,</li> Brotli: 204.18 seconds,</li> Zstandard: 0.15 seconds.</li> </ul> LZ4 and Zstandard excel in speed, with LZ4 being slightly faster. Zlib offers a middle ground, but Brotli, despite its high compression efficiency, takes significantly longer, which could be a drawback for real-time applications.</p> Decompression Time</h2> Decompression time is vital for applications where data needs to be rapidly restored to its original state:</p> Zlib: 11.99 seconds,</li> LZ4: 0.46 seconds,</li> Brotli: 0.99 seconds,</li> Zstandard: 0.46 seconds.</li> </ul> Again, LZ4 and Zstandard show excellent performance, both under half a second. Brotli presents a decent time despite its lengthy compression time, while zlib lags behind in this aspect.</p> Conclusion</h2> Each library has its strengths and weaknesses:</p> Brotli is your go-to for maximum compression but at the cost of time, making it suitable for applications where compression time is less critical.</li> Zstandard offers a great balance between compression ratio and speed, recommended for a wide range of applications.</li> LZ4 shines in speed, ideal for scenarios requiring rapid data processing.</li> Zlib provides moderate performance across the board.</li> </ul> Choosing the right library depends on your specific needs, whether it's speed, space, or a balance of both. This experiment provides a clear picture of what to expect from these libraries, helping you make an informed decision based on your application's requirements.</p> Building an Efficient WebSocket Server with Actix Actors and Protobuf Unknown — Thu, 29 Feb 2024 00:00:00 +0000 In the modern era of real-time web applications, WebSocket has emerged as a cornerstone technology enabling interactive communication sessions between a user's browser and a server. With the need for high performance, reliability, and scalability in these applications, choosing the right technology stack is paramount. In this context, Rust's Actix framework, combined with the efficiency of Protobuf for message serialization, offers a compelling choice over other popular technologies like Python, Node.js, and Go. Let's delve into how to implement a WebSocket server using Actix actors and why this combination might just be the superior choice for your next project.</p> Why Choose Rust's Actix?</h2> Before we jump into the implementation details, let's understand why Rust, particularly with its Actix framework, stands out:</p> Performance</h2> Rust's zero-cost abstractions and ownership model ensure memory safety without the need for a garbage collector, leading to blazing-fast performance. Actix further leverages Rust's async capabilities, making it incredibly efficient for IO-bound tasks common in WebSocket communication.</p> Reliability</h2> Rust's compile-time checks eliminate a whole class of runtime errors, making your WebSocket server more reliable. Actix actors encapsulate state and behavior, reducing shared state and side effects, which are often sources of bugs in concurrent systems.</p> Scalability</h2> Actix's actor model provides a natural way to build concurrent and distributed systems. Each actor runs independently, allowing for easy scaling across threads or even network boundaries.</p> Implementing WebSocket Server with Actix</h2> Setting up a WebSocket server in Actix is straightforward. Here's a simplified version to get us started:</p> use actix::prelude::*; use actix_web::{web, App, HttpServer, HttpRequest, HttpResponse}; use actix_web_actors::ws; struct MyWs; impl Actor for MyWs { type Context = ws::WebsocketContext<Self>; } impl StreamHandler<Result<ws::Message, ws::ProtocolError>> for MyWs { fn handle(&mut self, msg: Result<ws::Message, ws::ProtocolError>, ctx: &mut Self::Context) { if let Ok(ws::Message::Binary(bin)) = msg { // handle binary (Protobuf) messages here } } } async fn ws_index(r: HttpRequest, stream: web::Payload) -> Result<HttpResponse, actix_web::Error> { ws::start(MyWs {}, &r, stream) } #[actix_web::main] async fn main() -> std::io::Result<()> { HttpServer::new(|| { App::new().route("/ws/", web::get().to(ws_index)) }) .bind("127.0.0.1:8080")? .run() .await } </code></pre> Efficient Message Parsing with Protobuf</h2> For fast and efficient parsing of message events, we integrate Protobuf. Protobuf not only reduces the payload size significantly compared to JSON but also offers a structured way to define and parse your data.</p> Define your Protobuf messages</strong>: First, you'll need .proto</code> files that define the structure of your messages.</li> Generate Rust code</strong>: Use prost-build</code> in your build.rs</code> to generate Rust code from your .proto</code> files.</li> Serialize/Deserialize with Protobuf</strong>: Modify the WebSocket message handling in MyWs</code> to serialize and deserialize messages using the generated Protobuf code.</li> </ol> Comparing with Python, Node.js, and Go</h2> While Python and Node.js are immensely popular for their rapid development capabilities, their performance, particularly in CPU-bound tasks and highly concurrent environments, doesn't match Rust's. Python's dynamic nature and Node.js's single-threaded event loop can be limiting for real-time, high-throughput applications.</p> Go, on the other hand, offers excellent concurrency support through goroutines and a performance profile that's generally better than Python or Node.js. However, Rust's zero-cost abstractions and lack of a garbage collector can lead to more predictable performance under load, making it a strong competitor, especially in scenarios where latency and throughput are critical.</p> Conclusion</h2> By leveraging Actix and Protobuf, you can build a WebSocket server that stands out in terms of performance, reliability, and scalability. This powerful combination not only ensures efficient message parsing with minimal overhead but also benefits from Rust's safety and concurrency model, making it an excellent choice over Python, Node.js, and Go for your next real-time application.</p> The journey doesn't stop here, though. Experimenting with Actix's full potential, exploring Protobuf's capabilities, and benchmarking your server against other technologies will provide deeper insights and help tailor your application to your specific needs. Happy coding!</p> The Feathered Bond: Insights from Raising Birds of All Sizes Unknown — Sat, 24 Feb 2024 00:00:00 +0000 In the gentle flutter of wings and the soft chirping at dawn, my journey with birds began — not as an expert avian enthusiast but as a curious soul drawn to these vibrant creatures. Over the years, my aviary have been a sanctuary to Sun Conures, Green Cheek Conures, parakeets, lovebirds, and pigeons. Each bird, with its unique chirp and character, has taught me the profound intricacies of avian bonding. The most striking lesson? The pivotal role of hand-raising and the surprising impact of a bird's size on its ability to bond with humans.</p> Hand-Raised Hearts</h2> Half of my feathered friends came to me as babies, their tiny hearts beating fast, their eyes wide with the innocence of youth. These hand-raised birds grew up with human touch as a cornerstone of their existence. The comfort they find in human presence, the way they seek warmth from my hands, speaks volumes of the bond formed during those early days. The process wasn't merely about feeding; it was about building trust, a bridge between species, one gentle interaction at a time.</p> The Adult Adoption Challenge</h2> The other half of my avian family came as adults, each with a history I could only guess at. These birds, despite my best efforts, maintained a cautious distance. Most would fly away as I entered the aviary, an invisible barrier between us, save for my adult-purchased Sun Conures, who, in their own time, chose to bridge that gap. This stark difference in bonding led me to ponder the nuances of avian trust and the potential hurdles in bonding with birds that have not been hand-raised.</p> Size Matters in Avian Bonds</h2> An unexpected pattern emerged over time — the larger the bird, the more likely it was to bond well with humans. My smaller birds, like the parakeets and lovebirds, remained distant, their trust harder to earn. In contrast, my Sun Conures, despite their later introduction to human interaction, showed remarkable openness. This observation led me down a path of inquiry and speculation: Does the size of a bird inherently influence its capacity for human connection, or are other factors at play?</p> The Sun-Conure Exception</h2> My Sun Conures, adopted as adults, defy the norm. Unlike their smaller counterparts, they exhibit a curiosity and warmth towards me that belies their later start in human interaction. This exception to the rule has been a source of both joy and curiosity. It prompts questions about the roles of species traits, individual personalities, and perhaps the nuances of my approach to their care.</p> Fostering Feathers of Trust</h2> Bonding with both hand-raised and adult birds has been a journey of patience, understanding, and adaptation. For those looking to forge a bond with their feathered friends, here are a few tips:</p> Patience is Key</strong>: Bonding takes time, especially with adult-adopted birds. Respect their pace.</li> Consistency in Care</strong>: Regular, gentle interactions can build trust, even if progress seems slow.</li> Understanding Individual Needs</strong>: Each bird is unique. Tailor your approach to their personality and comfort level.</li> </ul> Reflecting on the Avian Connection</h2> My experiences have taught me that the essence of bonding with birds lies in understanding and respecting their individuality. Whether hand-raised or adopted as adults, small or large, each bird offers a unique window into the world of avian-human relationships. The journey of bonding with these magnificent creatures is a testament to the power of patience, empathy, and love.</p> Join the Flock</h2> I invite you to share your own stories of avian companionship. Have you noticed a correlation between a bird's size and its ability to bond with humans? What has been your experience in fostering a connection with hand-raised versus adult-adopted birds? Let's continue the conversation in the comments below.</p> Looking for a new Role Unknown — Fri, 09 Sep 2022 00:00:00 +0000 After 3.8 years of working at Reckonsys as VP of Engneering, I am looking for a new role now. My notice period at Reckonsys ended on August 31, 2022. One of the main problems I am facing has to do with carrying the title "VPE". People assume that I have taken up managerial position. But this if from from the truth. Despite being a VPE, 70% of my time has been invested in hands-on coding. To be honest, title "VPE" is wildly inaccurate for what I have been doing at Reckonsys. I would say only about 20% of time was spent doing what and Actual VPE would do. A glorified Tech Lead that took on part-time VPE</strong> role would be more accurate :P.</p> All that being said, I am currently looking for Python/Rust roles where I can be hands-on at least 70%</strong> of my time. More precisely, I love building micro-services</strong>, Setting up CI/CD</strong> or DevOps</strong> pipelines, scale</strong> said service and doing a bit of SRE</strong>.</p> I usually spend most of my time building python (micro)services. But I also have built production services with JavaScript (Browser + Node), C, Go, PHP, Ruby, Objective-C, C# & Java. I have worked on all levels of a typical web application stack. I architect, code, deploy, (auto)scale, and maintain server-side (.py, .rs), browser-side (.js, .ts, .wasm) & infra (terraform, pulumi, CDK) code. Here is a non-exhaustive list of things that I built in the past:</p> GraphQL, RPC & RESTful microservices with Python</li> JavaScript frontends with Ember & React</li> DevOps pipelines</li> E2E Test Automation Framework (Horizontal & Vertical) with RobotFramework</li> Native iOS apps</li> Android and iOS device farms</li> RaspberryPi bitcoin mining farm with custom ASIC chips</li> Scaled a couple of high-traffic web applications</li> Deploy services on AWS, Azure, and Google Cloud (Fabric, docker, AWS copilot, Kubernetes, etc.)</li> Small tech teams (5–15 engineers)</li> Setting up E2E engineering processes for the entire organization</li> Can design and develop resource-efficient, reliable, maintainable, secure & scalable products</li> A bunch of open-source tools that are useful for web development.</li> Contributed to a bunch of 3rd party open-source projects</li> Reverse Engineered and "hacked" a bunch of mobile applications for fun</li> Hobby IoT & robotic projects (very basic stuff)</li> </ul> Lately, I have developed a keen interest in Rust programming language, WebAssembly & Distributed software.</p> I would appreciate your support. Thank you in advance for any connections, advice, or opportunities you can offer.</p> Hi, I'm dhilipsiva Unknown — Mon, 19 Jun 2017 00:00:00 +0000 I have been coding for 10 years.</p> You can find me on GitHub as dhilipsiva</a></p> I live in Bangalore.</p> I work for Appknox</p> I mostly program in these languages: Python, JavaScript, Objective-C.</p> I am currently learning more about Machine Learning.</p> Nice to meet you.</p> Architecture at AppKnox Unknown — Thu, 10 Dec 2015 00:00:00 +0000 What does Appknox do?</h3> Appknox helps developers and enterprises to uncover and fix security loopholes in mobile applications.</p> Securing your app is as simple as submitting your store link / uploading your app. We scan for security vulnerabilities and report back to you with vulnerabilities.</p> Primary Languages</h3> Python & Shell for the Back-end</li> CoffeeScript and LESS for Front-end</li> </ol> Our Stack</h3> Django</li> Postgres (Migrated from MySQL)</li> RabbitMQ</li> Celery</li> Redis</li> Memcached</li> Varnish</li> Nginx</li> Ember</li> </ol> (The original post included an architecture diagram; it didn't survive the archive.)</em></p> How it works?</h3> Our back-end architecture consists of 3 subsystems: Client, Data and Workers.</p> Client Subsystem</h4> The client subsystem consists of two different load-balanced, auto-scaling App & Socket Servers. This is where all user-interactions takes place. We took much care not to have any blocking calls here to ensure lowest possible latency.</p> App Server</strong>: Each App server is a single Compute unit loaded with Nginx and Django-gunicorn server, managed by supervisord. User requests are served here. When a user submits the url their app, we submit it to the RabbitMQ download queue and immediately let user know that the URL has been submitted. In case of uploading any app, a signed-url is fetched from server. The browser uploads data directly to the S3 with this signed-url and notifies the app server when it is done.</p> Socket server</strong>: Each socket server is a single compute unit loaded with Nginx and a node (socket-io) server. This server uses Redis as its adapter. And yes, of course, this is used for real-time updates.</p> Data subsystem</h4> This system is used for data storage, queuing and pub/sub. Which is also responsible for a decoupled architecture.</p> Database Cluster</strong>: We use Postgres. It goes without saying that it consists of a Write-Heavy master and few Read-Heavy replicas.</p> RabbitMQ</strong>: A broker to our celery workers. We have different queues for different workers. Mainly download, validate, upload, analyse, report, mail and bot. The web server puts data into queue, the celery workers pick it up and run it.</p> Redis</strong>: This acts a adapters to socket-io servers. When ever we want to notify user an update from any of our workers, we publish it to Redis, which in turn will notify all users trough Socket.IO.</p> Worker Subsystem</h4> This is where all the heavy lifting works are done. All the workers gets tasks from RabbitMQ and Published updates to users thorough Redis.</p> Static Scanners</strong>: This is an auto-scaling Compute unit group. Each unit consists of 4–5 celery workers. Each celery worker scans single app at a time.</p> Other tasks</strong>: This is an auto-scaling Compute unit group. Each unit consists of 4–5 celery workers which does various tasks like download apps from stores, generating report pdf, uploading report pdf, sending emails, etc.</p> Dynamic Scanning</strong>: This is platform-specific. Each Android dynamic scanner is a On-Demand Compute instance that has android emulator (With SDKs) and a script that captures data. This emulator is shown on a canvas in the browser for user to interact. Each iOS scanner is in a managed Mac-Mini farm that has scripts and simulators supporting the iOS platform.</p> Reasons for choosing the stack</h3> We chose Python because the primary libraries that we use to scan applications is in python. Also, we love python more than any other languages that we know.</p> We chose Django because it embraces modularity.</p> Ember — We think that this is the most awesome Front-end framework that is out there. Yes, the learning curve is too steep than any-other, but once you climb that steep mountain, you will absolutely love ember. It is very opinionated. So as long as you stick to its conventions, you write less to do more.</p> Postgres — Originally, we chose MySQL because it was de-facto. After Oracle purchased Sun Microsystems (Parent company of MySQL), MySQL became stagnant. I guess we all expected it. So we dided to use MariaDB [A fork of MySQL] maintained by community. Later, we required persistent key-value stores a bit, which is offered out of the box by Postgres. It plays really well with Python. We use UUIDs as primary keys which is a native data type in Postgres. Also, the uuis-ossp module provided functions to generate and manipulate UUIDs at the Database level, rather than creating them at application level, which was costlier. So we switched to Postgres.</p> And the rest are de-facto. RabbitMQ for Task Queues. Celery for Task Management. Redis for Pub/Sub. Memcached & Varnish for caching.</p> Things that didn't go as expected</h3> One of the things that didn't go as expected is scaling sockets. We were using Django-socket.io initially. We realized that this couldn't be scaled to multiple servers. So we wrote that as a separate node module. We used node's socket-io library that supported Redis-adapter. Clients are connected to the node's socket server. So we now publish to Redis from our python code. Node will just push the notifications to clients. This can be scaled independently of the app-server that acts as a JSON endpoint to the clients.</p> Notable stuff about our stack</h3> We love modular design. We went to modularize stuff so far that we de-coupled our front-end from our back-end. Yes, you read it right. All the HTML, CoffeeScript and LESS code is developed independently of the back-end. Front-end development does not require server to be running. We rely on front-end fixtures for fake data during development.</p> Our back-end is named Sherlock. We detect security vulnerabilities in mobile applications. So the name seemed apt. Sherlock is smart.</p> And our Front-end is named Irene. Remember Irene Adler? She is beautiful, colorful and tells our user's whats wrong.</p> And our Admin is named Hudson. Remember Mrs.Hudson? Sherlock's land-lady? Thinking of which we should have probably given a role to poor Dr.Watson. Maybe we will.</p> So Sherlock does not serve any HTML/CSS/JS files. I repeat, It does not serve ANY single static file / HTML file. Both sherlock and Irene are developed independently. Both have separate deployment process. Both have their own test-cases. We deploy sherlock to Compute instances and we deploy Irene to Google Cloud Storage.</p> The advantage of such architecture is that:</p> The Front End team can work independent of the back-end without stepping on each other toes.</li> The heavy lifting work like rendering pages on the server is taken off of server.</li> We can open-source the front-end code. Making it easy to hire front-end guys. Just ask them to fix a bug in the repo and they are hired. After all, front-end code can be read by anyone even if you don't open-source it right?</li> </ol> Our deployment process</h3> The code is auto-deployed from the master branch. We follow Vincent Driessen</a>'s Git branching model. Jenkins build commits to develop branch. If it succeeds we do another manual testing, just to be sure and merge it with master branch and it gets auto deployed.</p> Appropriate Number of Gunicorn Workers Unknown — Thu, 22 Oct 2015 00:00:00 +0000 To quote Gunicorn documentation</a>:</p> Generally we recommend (2 x $num_cores) + 1 as the number of workers to start off with. While not overly scientific, the formula is based on the assumption that for a given core, one worker will be reading or writing from the socket while the other worker is processing a request. Obviously, your particular hardware and application are going to affect the optimal number of workers. Our recommendation is to start with the above guess and tune using TTIN and TTOU signals while the application is under load.</p> </blockquote> Gunicorn docs suggests that 2n+1 (gunicorn -w <2n+1> myapp:wsgi</code>) is a good guess for number of workers (Yes, n = number of cores). I came up with a tiny shell script to apply this formula. All you need to do is this:</p> gunicorn -w $(( 2 * `cat /proc/cpuinfo | grep 'core id' | wc -l` + 1 )) myapp:wsgi </code></pre> Where the command</p> cat /proc/cpuinfo | grep 'core id' | wc -l </code></pre> will return the total number of actual CPU cores (n). So,</p> $(( 2 * `cat /proc/cpuinfo | grep 'core id' | wc -l` + 1 )) </code></pre> equates to 2n+1 formula.</p> This will apply 2n+1 formula to all the linux-based machines.</p> You can also achieve this with Gunicorn Configuration File</a> gunicorn.conf.py as follows:</p> import multiprocessing bind = "127.0.0.1:8000" workers = multiprocessing.cpu_count() * 2 + 1 </code></pre> This was an answer that I posted on StackOverflow</a>.</p> Python Libraries — Django, Twisted, Tornado, Flask, Cyclone and Pyramid Unknown — Sun, 19 May 2013 00:00:00 +0000 "Django</strong></a> is a high-level Python Web framework that encourages rapid development and clean, pragmatic design"</em>. If you are building something that is similar to a e-commerce site, then you should probably go with Django</code>. It will get your work done quick. You don't have to worry about too many technology choices. It provides everything thing you need from template engine to ORM</code>. It will be slightly opinionated about the way you structure your app, which is good If you ask me. And it has the strongest community of all the other libraries, which means easy help is available.</p> "Flask</strong></a> is a microframework for Python based on Werkzeug, Jinja 2 and good intentions"</em>. Beware - "microframework" may be misleading. This does not mean that Flask</code> is a half-baked library. This mean the core of Flask</code> is very, very simple. Unlike Django</code>, It will not make any Technology decisions for you. You are free to choose any template engine or ORM</code> that pleases you. Even though it comes with Jinja</code> template engine by default, you are always free to choose our own. As far as I know Flask</code> comes in handy for writing APIs endpoints (RESTful rervices</code>).</p> "Twisted</strong></a> is an event-driven networking engine written in python"</em>. This is a high-performance engine. The main reason for its speed is something called as deferred. Twisted</code> is built on top of deferred</code>. For those of you who don't know about deferred, it is the mechanism through with asynchronous architecture is achieved. Twisted</code> is very fast. But is not suitable for writing conventional WebApps. If you want to do something low-level networking stuff, Twisted</code> is your friend.</p> "Tornado</strong></a> is a Python web framework and asynchronous networking library, originally developed at FriendFeed</code>. By using non-blocking network I/O, Tornado</code> can scale to tens of thousands of open connections, making it ideal for long polling, WebSockets</code>, and other applications that require a long-lived connection to each user"</em>. Tornado</code> stands some where between Django</code> and Flask</code>. If you want to write something with Django</code> or Flask</code>, but if you need a better performance, you can opt for Tornado</code>. It can handle C10k problem very well if it is architected right.</p> "Cyclone</strong></a> is a web server framework for Python that implements the Tornado</code> API as a Twisted protocol"</em>. Now, what if you want something with Twisted</code>'s performance and easy to write conventional webapps? Say hello to Cyclone</code>. I would prefer Cyclone</code> over Tornado</code>. It has an API that is very similar to Tornado</code>. As a matter of fact, this is a fork of Tornado</code>. But the problem is it has relatively small community. Alexandre Fiori is the only main commiter to the repo.</p> "Pyramid</strong></a> is a general, open source, Python web application development framework. Its primary goal is to make it easier for a Python developer to create web applications"</em>. I haven't really used Pyramid</code>, but I went through the documentation. From what I understand, Pyramid</code> is very similar to Flask</code> and I think you can use Pyramid</code> wherever Flask</code> seems appropriate and vice-versa.</p> Want me to review any other library? Please leave a comment below.</p> Comments / Suggestions / Edits welcomed.</p> I answered a question on StackOverflow here</a></p>