In our twentieth issue of the Architects' Newsletter we are focusing on the topic of data engineering and streaming architecture. This topic has largely "crossed the chasm" in regard to the diffusion of innovation, and in our latest InfoQ Architecture and Design Trend report we have event-driven architecture in the early majority phase, and other related patterns and approaches in the early adopter phase. Understanding all the emerging patterns, antipatterns, and technologies is therefore essential for a software architect.

News

Google Expands Cloud BigTable Replication Capabilities Globally in Beta

Google announced the expansion of Cloud BigTable's replication capabilities in Beta, providing customers with the flexibility to make their data available across a region or worldwide. The enhancements will allow customers to create a replicated cluster in any zone at any time.

Comparable NoSQL services for Cloud Bigtable are Cosmos DB from Microsoft and DynamoDB from Amazon - all having replication capabilities. Moreover, Cosmos DB and DynamoDB offer similar capabilities as Cloud Bigtable, however there is a significant difference in supporting database models and APIs for accessing data. For example, AWS offer DynamoDB Streams for obtaining access to the underlying changes made to a table.

Patterns of Streaming Applications at Netflix

The recording of Monal Daxini's QCon SF talk "Patterns of Streaming Applications" is now available on InfoQ, and in it he presents a blueprint for streaming data architectures and a review of desirable features of a streaming engine. He also talks about streaming application patterns and anti-patterns, and use cases and concrete examples using Apache Flink.

The Whys and Hows of Database Streaming

In another QCon SF presentation recording, Joy Gao talks about how database streaming is essential to WePay's infrastructure and the many functions that database streaming serves. She provides information on how the database streaming infrastructure was created and managed so that others can leverage their work to develop their own database streaming solutions. She also reviews challenges faced with streaming peer-to-peer distributed databases.

The Importance of Event-First Thinking

For global businesses to meet today's architectural challenges with constant change and extreme scale, we need to go back to the basic principles of system design. The common element in the problems we face is the notion of events driving both actions and reactions, Neil Avery writes in a series of blog posts describing why events are important and the advantages of an event-first approach that also supports an evolutionary architecture.

SpringOne Platform InfoQ Videos on Streaming

A number of recorded streaming-themed sessions from SpringOne Platform are now available on InfoQ. In "Fast and Furious: Searching in a Distributed World with Highly Available Spring Data Redis", Julien Ruaux discusses the Redis Enterprise architecture and demos Redis clusters by building three microservices, performing full-text searches, and showing the results using Spring Boot Web and Angular.

In "Next Generation MongoDB: Sessions, Streams, Transactions" Christoph Strobl and Jeff Yemin discuss some of the data engineering features in latest MongoDB versions: sessions, change streams, retriable writes, reactive access and transactions. The talk, "Global Event Streams Made Simple with Spring Cloud Stream & Cloud Pub/Sub", features Artem Bilan and Kir Titievsky discussing the Google Cloud Pub/Sub binding for the Spring Cloud Stream framework.

Case Study

Grady Booch: AI today is a System Engineering Problem with AI Components.

InfoQ editor Michael Stiefel has written a summary of Grady Booch's QCon SF keynote. According to Booch, most current AI systems are about pattern matching of signals at the edge and inductive reasoning, not true artificial intelligence. During his second day keynote at the 2018 QCon San Francisco, "Building the Enchanted Land", he explained his view that AI today is a "system engineering problem with AI components."

True AI uses decision making and abductive reasoning, which allows those systems to reason and learn. Current artificial intelligence applications are far from being able to accomplish that; they are just components in larger systems.

Contemporary AI is not of recent origin, indeed many of the architectures and algorithms are decades old. The difference today is the abundance of computational power, and the existence of large bodies of tagged data. In fact, working with the data to understand it, and get it into the proper format will most often take more work than the building of the model.

Developing AI systems requires the same skills as classical software development: crisp abstractions, separation of concerns, balanced distribution of responsibilities, and an attempt to keep things simple. You want to grow your system through incremental and iterative releases.

Of course there are unresolved problems. One of them is how to bring together symbolic, connectionist, and quantum models of computation. Booch strongly disagrees with the views of the DeepMind community that thinks that AI and neural networks are going to be at the center of systems.

He called attention to Moravec's paradox. There are far more neurons in the human brain devoted to signal processing, the visual cortex, and the hearing cortex than there are in the decision making system. This kind of split appears in the hybrid systems of today. Artificial intelligence for the edge, symbolic systems for decisions processing, and conventional software around it.

As Alan Newell observed during one of the AI winters, "Computer technology offers the possibility of incorporating intelligent behavior in all the nooks and crannies of the world. With it, we can build an enchanted land." To Booch, software is the invisible language that whispers stories of possibility to the hardware. Today, software developers are the storytellers, the ones who are going to build the enchanted land.

This excerpt is based on a full article can be found on InfoQ "Grady Booch on the Future of AI". You can also hear an interview with Booch discussing AI in this episode of the InfoQ podcast.

To get notifications when InfoQ publishes content on these topics follow "Streaming" and "AI, ML and Data Engineering" on InfoQ.

Missed a newsletter? You can find all of the previous issues on InfoQ.

This edition of The Software Architects' Newsletter is brought to you by:

Only a few tickets left for QCon.ai, April 15-17, 2019!

We are only a few weeks away from QCon.ai, the applied AI and Software conference for developers. Over 90% of the speakers are confirmed. Each QCon.ai speaker is working on real-world AI and ML production projects. Learn from speakers driving innovation and change in AI and machine learning from companies including Google, Facebook, Stitch Fix, PayPal, Airbnb, Slack, and more.

Don't be left out, save your seat at this year’s conference. Register using the code infoqai19 to get an $80 discount!

InfoQ strives to facilitate the spread of knowledge and innovation within this space, and in this newsletter we aim to curate and summarise key learnings from news items, articles and presentations created by industry peers, both on InfoQ and across the web. We aim to keep readers informed and educated about emerging trends, peer-validated early adoption of technologies, and architectural best practices, and are always keen to receive feedback from our readers. We hope you find it useful, but if not you can unsubscribe using the link below.

Forwarded email? Subscribe and get your own copy.