How-To Tutorials

This year the JVM Language Summit 2019 was held on July 29th – 31st at Santa Clara, California. On the first day of the Summit, Oracle Java language architect Brian Goetz gave a talk on updates to OpenJDK Project Valhalla. He shared details on its progress, challenges being faced and what to expect from Project Valhalla in the future. He also talked about the significance of Project Valhalla’s LW2 phase which was released earlier last month.  OpenJDK Project Valhalla is now ready for developers to use for early-adopter experimentation in data structures and language runtimes, concluded Goetz. The main goal of OpenJDK Project Valhalla is to reboot the Java Virtual Machine (JVM) relationship with data and memory and in particular to enable denser and flatter layouts of object graphs in memory. The major restriction in the development of object layout has been the object identity. Object identity enables mutability, layout polymorphism and locking among others. As all objects do not need object identity and it would be impractical to determine whether an identity is relevant or not, Goetz expects programmers to inform about the whereabouts of a class such that it will make it easier to make a broader range of assumptions about it. Who cares about Value types? Goetz believes that value types are important for many applications and writers who desire a better control of memory layout and like to use memory very wisely. He says that library writers would always prefer Value types as it allows them to use all the traditional abstracts without paying the runtime cost of taking an extra indirection every time somebody uses a particular abstraction. Thus library classes like optional or cursors or better numerix do not have to pay the object tax.  Similarly, compiler writers of non-Java languages use Value types as an efficient substrate for language features like tuples, multiple return, built-in numeric types and wrapped native resources. Thus both library writers and compiler writers and their users pay the object tax.  Value types, in a nutshell, can help programmers make their code run faster.  Erased and specialized generics Currently, OpenJDK Project Valhalla uses erased generics and will eventually have specialized generics. In an erased generics, Valhalla uses the knowable type convention where the erased list of values can be called as Foo<V?>. This will also be moved to specialized generics later on. He also adds that this syntax cannot be used as of now, as the Valhalla team still does not have existing utterances of Foo for spontaneously changing their meaning. Goetz hopes that the migration of generic classes like Array List<T> to specialized generics would be painless.  New top types Project Valhalla needs new top types RefObject and ValObject for references and values as types are used to indicate a programmer’s intent. It helps the object model reflect the new reality, as everything is an object, but every object does not need an identity. There are many benefits of implementing ref-ness and val-ness into the type system such as: Dynamically ask x instanceof Ref object Statically constrain method parameters or return values Restrict type parameters Natural place to hang ref- or val-specific behavior Nullity Nullity is labelled as one of the most controversial issues in Valhalla. As many values use all their bit patterns, Nullity interferes with a number of useful optimizations. On the other hand, if some types are migrated towards values, the existing code will assume nullability. Nullity is expected to be a focus of the L3 investigation. What to expect next in Project Valhalla Lastly, Goetz announces that developers building data structures or compiler runtime libraries can start using Project Valhalla. He also adds that the Project Valhalla team is working hard to validate the current programming model by working on quantifying the costs of equality, covariance, etc and is trying to better the user control experience.  Goetz concluded by stating that OpenJDK Project Valhalla is at an inflection point and is trying to figure out Nullity, Migration, specialized generics and support for Graal in the future builds. You can watch the full talk of Brian Goetz for more details. Getting started with Z Garbage Collector (ZGC) in Java 11 [Tutorial] Storm 2.0.0 releases with Java enabled architecture, new core and streams API, and more Brian Goetz on Java futures at FOSDEM 2019

Last month, at USENIX ATC 2019, many systems researchers presented their work on topics including real-world deployed systems, runtimes, big-data programming models, security, virtualization, and much more. This year it happened from 10-12 July at Renton, WA, USA. The USENIX Annual Technical Conference (ATC) is considered to be one of the most prestigious systems research conferences. It covers all practical facets of systems software and aims to improve and further the knowledge of computing systems of all scales. Along with providing a platform to showcase cutting-edge systems research, it also allows researchers to gain insight into fields like virtualization, system management and troubleshooting, cloud and edge computing, security, and more. Here are some of the remarkable papers presented at this event: Captive - a retargetable system-level DBT hypervisor By: Tom Spink, Harry Wagstaff, and Björn Franke from the University of Edinburgh Why Captive is needed To boot an operating system and execute programs compiled for an Instruction Set Architecture (ISA) other than the host machine, system-level Dynamic Binary Translation (DBT) is used. DBT is a process of translating code for one ISA to another on the fly. Due to their performance-critical nature, DBT frameworks are generally hardcoded and heavily optimized for both their guest and host ISAs. Though this ensures performance gains, it poses high engineering costs for supporting a new architecture or extending an existing one. How Captive works The researchers have devised a novel, retargetable system-level DBT hypervisor called Captive. It includes guest specific modules generated from high-level guest machine specifications, which simplifies retargeting of the DBT and relieves users from low-level implementation effort. Captive enforces aggressive optimizations by combining the offline optimizations of the architecture model with online optimizations performed within the generated Just-In-Time compiler. It reduces the compilation overheads while providing high code quality. Additionally, it operates in a virtual bare-metal environment provided by a VM hypervisor. This allows you to fully exploit the underlying host architecture, especially the system-related and privileged features not accessible to other DBT systems operating as user processes. Here’s a diagram depicting how it works: Source: Usenix ATC The researchers evaluated the DBT based on both targeted micro-benchmarks and standard application benchmarks. They also compared it with the de-facto standard Qemu DBT system. The evaluation revealed that Captive delivers an average speedup of 2.21x over Qemu across SPEC CPU2006 integer benchmarks. In the case of floating-point applications, it shows further speedup reaching a 6.49x average. It also significantly reduces the effort required to support a new ISA, while delivering outstanding performance. To know more about Captive, check out this USENIX ATC '19 lightning talk by the authors: https://www.youtube.com/watch?v=syhmsBA6OpY SILK - a new open-source key-value store derived from RocksDB, designed to prevent latency spikes By: Oana Balmau, Florin Dinu, and Willy Zwaenepoel, University of Sydney; Karan Gupta and Ravishankar Chandhiramoorthi, Nutanix, Inc.; Diego Didona, IBM Research–Zurich Why SILK is needed Latency-critical applications demand data platforms that can provide low latency and predictable throughput. Log-structured merge key-value stores (LSM KVs) were designed to handle such write-heavy workloads and large scale data where working set does not fit in the main memory. Some of the common LSM KVs are RocksDB, LevelD, and Cassandra that are widely adopted in production environments and claim to optimize the heavy workload. Despite these claims, the researchers show that tail latencies in state-of-the-art LSM KVs can be quite poor, particularly in the case of heavy and variable client write loads. How SILK works To address the aforementioned limitations, the researchers have come up with the notion of an I/O scheduler for LSM KVs, which aims to achieve the following three goals: Opportunistically allocating I/O bandwidth to internal operations Prioritizing internal operations at the lower levels of the tree. Preempting compactions. This notion of I/O scheduler is implemented in SILK, a new open-source KV store derived from RocksDB. It is designed to prevent client request latency spikes. It uses this I/O scheduler to manage external client load and internal LSM maintenance work. It was tested on a production workload and synthetic benchmarks and was able to achieve up to two orders of magnitude lower 99th percentile latencies than RocksDB and TRIAD. To know more about SILK, check out this USENIX ATC '19 lightning talk by the authors: https://www.youtube.com/watch?v=6kPYDFT1q8w Transactuations and its implementation, Relacs for building reliable IoT applications By: Aritra Sengupta, Tanakorn Leesatapornwongsa, and Masoud Saeida Ardekani, Samsung Research; Cesar A. Stuardo, University of Chicago Why transactuations and the Relacs runtime are needed IoT applications are responsible for reading sensors, executing application logic, and taking action with actuators accordingly. One of the challenges developers face while building an IoT application is ensuring its correctness and reliability. Though current solutions do offer simple abstractions for reading and actuating, they lack high-level abstractions for writing reliable and fault-tolerant applications. Not properly handling failures can lead to inconsistencies between the physical and application state. How transactuations and Relacs work In this paper, the researchers introduced “transactuations”, which are similar to database transactions. These abstract the complexity of handling various failures and make it easy to maintain soft states so that they are consistent with respect to reads and writes to hard states. You need to specify dependencies among operations on soft and hard states along with sensing or actuating policy that specifies the conditions under which soft states can commit despite failures. Transactuation will then be responsible for preserving this dependence even in cases of hardware and communication failures and ensure isolation among transactuations that are executing concurrently. The researchers have also introduced Relacs, a runtime system that implements the abstraction for a smart home platform. It first transforms an application into a serverless function and executes the application in the cloud while enforcing transactuation specific semantics. The researchers further showed that transactuations are an effective solution for building reliable IoT applications. Using them also significantly reduces lines of code compared to manually handling failures. The Relacs runtime also guarantees reliable execution of transactuations while imposing reasonable overheads over a baseline that does not provide consistency between operations on hard states and soft states. To know more about transactuations, check out this USENIX ATC '19 lightning talk by the authors: https://www.youtube.com/watch?v=aBUYHWsWrhE Browsix-Wasm - Run unmodified WebAssembly-compiled Unix applications inside the browser in a performant way By: Abhinav Jangda, Bobby Powers, Emery D. Berger, and Arjun Guha, University of Massachusetts Amherst Why Browsix-Wasm is needed Major browsers today including Mozilla, Chrome, Safari, and Edge support WebAssembly, a small binary format that promises to bring near-native performance to the web. It serves as a portable compilation target for high-level languages like C, C++, and Rust. One of the key goals of WebAssembly is performance parity with native code. The paper that introduced WebAssembly showed that its performance is competitive with native code. However, the evaluation was limited to a suite of scientific kernels rather than full applications with each consisting of about 100 lines of code. The researchers conducted a comprehensive performance analysis using the established SPEC CPU benchmark suite of large programs. However, using such suites also pose a challenge, that is, currently it is not possible to simply compile a sophisticated native program to WebAssembly. These need operating system services such as a filesystem, synchronous I/O, and processes, which WebAssembly and the browser do not provide. How Browsix-Wasm works As a solution to this challenge, the researchers have built Browsix-Wasm, an extension to Browsix that allows running unmodified WebAssembly-compiled Unix applications directly inside the browser. They used Browsix-Wasm to conduct the very first large-scale evaluation of the performance of WebAssembly vs. native. The evaluation results show a substantial performance gap across the SPEC CPU suite of benchmarks. The applications compiled to WebAssembly were slower by an average of 45% (Firefox) to 55% (Chrome), with peak slowdowns of 2.08$\times$ (Firefox) and 2.5$\times$ (Chrome). Some of the reasons behind this performance degradation were missing optimizations and code generation issues. Here’s a chart showing the comparison between the performance analysis done on the basis of PolyBenchC (previous work) and SPEC CPU benchmarks. Source: Usenix ATC To know more about Browsix-Wasm, check out this USENIX ATC '19 lightning talk by the authors: https://youtu.be/-6HoiVWiHiw Zanzibar - Google’s global authorization system By: Ruoming Pang, Ramon Caceres, Mike Burrows, Zhifeng Chen, Pratik Dave, Nathan Germer, Alexander Golynski, Kevin Graney, and Nina Kang, Jeffrey L. Korn, Christopher D. Richards and Mengzhi Wang, Google; Lea Kissner, Humu, Inc.; Abhishek Parmar, Carbon, Inc. Why Zanzibar is needed Every day online interactions involve the exchange of a lot of personal information. These interactions are authorized to ensure that a user has permission to perform an operation on a digital object. For instance, we have several web-based photo storage services that allow users to share a few photos with friends while keeping other photos private. These services must have checks in place to ensure that photos are shared before another user can view them. There are already many ways of authorization and developers constantly work on making them more robust to guarantee online privacy. How Zanzibar works The researchers have come up with Zanzibar, a system that allows you to store permissions and perform authorization checks based on the stored permissions. Many Google services use it including Calendar, Cloud, Drive, Maps, Photos, and YouTube. Zanzibar takes up two roles: A storage system for Access Control Lists (ACLs) and groups. An authorization engine that interprets permissions. It provides a uniform data model and language to define a wide range of access control policies. While making authorization decisions it takes into account the causal ordering of user actions to provide external consistency amid changes to access control lists and object contents. Here’s a diagram depicting its architecture: Source: Usenix ATC Zanzibar is scalable to trillions of access control lists and millions of authorization requests per second to support services. In more than 3 years of production use at Google, it has maintained 95th-percentile latency of less than 10 milliseconds and availability of greater than 99.999%. To know more about Zanzibar, check out this USENIX ATC '19 lightning talk by the authors: https://www.youtube.com/watch?v=cJ334qJ0jBI These were some of the papers presented at USENIX ATC 2019. You can find other research papers on its official website. International cybercriminals exploited Citrix internal systems for six months using password spraying technique Google Cloud and Nvidia Tesla set new AI training records with MLPerf benchmark results “Don’t break your users and create a community culture”, says Linus Torvalds, Creator of Linux, at KubeCon + CloudNativeCon + Open Source Summit China 2019  

Every year a new edition of the ECMAScript (ES) scripting-language specification standard comes out. This year it is its tenth edition, also known as ES2019 or ES10. The feature set of ES2019 got finalized earlier this year and was published last month. Some of the exciting features this specification brings are Object.fromEntries(), trimStart(), trimEnd(), flat(), flatMap(), description property for symbol objects, optional catch binding, and more. These features have also landed in the latest versions of Firefox and Chrome for developers to try out. Let’s take a look at some of the features in ES2019: Object.fromEntries() In JavaScript, you can easily convert objects into arrays with the Object.entries() method that was introduced in the ES2017 standard. ES2019 introduces the Object.fromEntries() method that enables you to do exactly the opposite. Similar to the dict() function in Python, this method allows you to transform a list of key-value pairs into an object. Array.prototype.flat() and Array.prototype.flatMap() The method Array.prototype.flatten() was renamed to Array.prototype.flat() method in ES2019 after last year it ended up breaking MooTools' implementation of it. It recursively flattens an array up to the specified depth, which defaults to 1. The second method, ‘Array.prototype.flatMap’ performs the mapping of each element and then flattens the result into a new array. trimStart() and trimEnd() The purpose of the new trimStart() and trimEnd() methods proposed in ES2019 is same as the trimLeft() and trimRight() methods. While trimStart() is used to remove whitespace from the beginning of a string, trimEnd() is used to remove whitespace characters from the end of a string. These are introduced to maintain consistency with the padStart/padEnd the standard functions. To maintain web compatibility trimLeft() and trimRight() will be their aliases. Optional catch binding In JavaScript, it is mandatory to specify the catch() parameter when using try...catch, no matter whether you use it or not. However, there are a few use cases where you wouldn’t want to use the parameter or catch binding. Axel Rauschmayer, the author of JavaScript for impatient programmers (ES1–ES2019), lists the following two: If you want to completely ignore the error. You don’t care about the error or you already know what it will be, but you do want to react to it. This new proposal allows you to completely omit the unused catch binding without any syntax errors. Function.toString() Earlier, when you called the toString() method on a function it used to strip all the whitespaces, newlines, and comments from the source code. Now, it will return the function source code exactly as it was defined. Description property for Symbol objects ES2019 introduces a new read-only ‘description’ property for Symbol objects. You can add it to a Symbol object to return a string containing its description for debugging purposes. Well-formed JSON.stringify() According to a JSON RFC, JSON text when shared “outside the scope of a closed ecosystem” should be encoded using UTF-8. However, JSON.Stringify() can sometimes return strings and code points, particularly, the surrogate range (U+D800—U+DFFF), that cannot be represented in UTF-8. This ES2019 proposal prevents JSON.stringify() from returning such ill-formed Unicode strings. Many developers are excited about these new ES2019 proposals. A user on Hacker News commented, “That array.flat() and array.flatMap() stuff is great to see. Always having to rely on lodash and friends to do that type of work. Exciting to see how JS is evolving.” Another user added, “Object.fromEntries will be super useful, surprised it’s taken this long to become a native feature.” Others are waiting for the pattern matching and optional chaining proposals to reach the stage 4 of TC39 process, “Now if we could just get pattern matching and optional chaining, that would really elevate things.” These were some of the features introduced in ES2019. To know more, check out the specification published on the ECMA International website. Introducing QuickJS, a small and easily embeddable JavaScript engine Firefox 67 will come with faster and reliable JavaScript debugging tools Introducing Node.js 12 with V8 JavaScript engine, improved worker threads, and much more  

Yesterday, two members of the Google Project Zero team revealed about six “interactionless” security bugs that can affect iOS by exploiting the iMessage Client. Four of these bugs can execute malicious code on a remote iOS device, without any prior user interaction. Apple released fixes for these bugs in the iOS 12.4 update on July 22. The two Project Zero researchers, Natalie Silvanovich and Samuel Groß, published details and demo proof-of-concept only for five out of the six vulnerabilities. Details of one of the "interactionless" vulnerabilities have been kept private because Apple's iOS 12.4 patch did not completely resolve the bug, according to Natalie Silvanovich. https://twitter.com/natashenka/status/1155941211275956226 4 bugs can perform an RCE via a malformed message Bugs with vulnerability IDs, CVE-2019-8647, CVE-2019-8660, CVE-2019-8662, CVE-2019-8641 (the one whose details are kept private), can execute malicious code on a remote iOS device. The attacker has to simply send a malformed message to the victim’s phone. Once the user opens the message and views it, the malicious code will automatically execute without the user knowing about it. 2 bugs can leak user’s on-device data to a remote device The other two bugs, CVE-2019-8624 and CVE-2019-8646, allow an attacker to leak data from a user’s device memory and read files off a remote device. This execution too can happen without the user knowing. “Apple's own notes about iOS 12.4 indicate that the unfixed flaw could give hackers a means to crash an app or execute commands of their own on recent iPhones, iPads and iPod Touches if they were able to discover it”, BBC reports. Silvanovich will talk about these remote and interactionless iPhone vulnerabilities at this year’s Black Hat security conference held at Las Vegas from August 3 - 8. An abstract of her talk reads, “There have been rumors of remote vulnerabilities requiring no user interaction being used to attack the iPhone, but limited information is available about the technical aspects of these attacks on modern devices.” Her presentation will explore “the remote, interaction-less attack surface of iOS. It discusses the potential for vulnerabilities in SMS, MMS, Visual Voicemail, iMessage and Mail, and explains how to set up tooling to test these components. It also includes two examples of vulnerabilities discovered using these methods." According to ZDNet, “When sold on the exploit market, vulnerabilities like these can bring a bug hunter well over $1 million, according to a price chart published by Zerodium. It wouldn't be an exaggeration to say that Silvanovich just published details about exploits worth well over $5 million, and most likely valued at around $10 million”. For iOS users who haven’t yet updated the latest version, it is advisable to install the iOS 12.4 release without any delay. Early this month, the Google Project Zero team revealed a bug in Apple’s iMessage that bricks iPhone causing a repetitive crash and respawn operations. This bug was patched in iOS 12.3 update. To know more about these five vulnerabilities in detail, visit the Google Project Zero bug report page. Stripe’s API degradation RCA found unforeseen interaction of database bugs and a config change led to cascading failure across critical services Azure DevOps report: How a bug caused ‘sqlite3 for Python’ to go missing from Linux images Is the Npm 6.9.1 bug a symptom of the organization’s cultural problems?

Update: On 28th August, an indictment was filed in a US federal district court, which mentioned Thompson allegedly hacked and stole information from an additional 30 AWS-hosted organizations and will face computer abuse charges. Capital One Financial Corp., one of the largest banks in the United States, has been subject to a massive data breach affecting 100 million customers in the U.S and an additional 6 million in Canada. Capital One said the hacker exploited a configuration vulnerability in its firewall that allowed access to the data. In its official statement released yesterday, Capital One revealed that on July 19, it determined an "unauthorized access by an outside individual who obtained certain types of personal information relating to people who had applied for its credit card products and to Capital One credit card customers." Paige A. Thompson, 33, the alleged hacker who broke into Capital One server, was arrested yesterday and appeared in federal court in Seattle. She was an ex-employee from Amazon's Cloud service (AWS), Amazon confirms. The Capital One hacker, an ex-AWS employee, “left a trail online for investigators to follow” FBI Special Agent Joel Martini wrote in a criminal complaint filed on Monday that a “GitHub account belonging to Thompson showed that, earlier this year, someone exploited a firewall vulnerability in Capital One’s network that allowed an attacker to execute a series of commands on the bank’s servers”, according to Ars Technica. IP addresses and other evidence ultimately showed that Thompson was the person who exploited the vulnerability and posted the data to Github, Martini said. “Thompson allegedly used a VPN from IPredator and Tor in an attempt to cover her tracks. At the same time, Martini said that much of the evidence tying her to the intrusion came directly from things she posted to social media or put in direct messages”, Ars Technica reports. On  July 17, a tipster wrote to a Capital One security hotline, warning that some of the bank’s data appeared to have been “leaked,” the criminal complaint said. According to The New York Times, Thompson “left a trail online for investigators to follow as she boasted about the hacking, according to court documents in Seattle”. She is listed as the organizer of a group on Meetup, a social network, called Seattle Warez Kiddies, a gathering for “anybody with an appreciation for distributed systems, programming, hacking, cracking.” The F.B.I. noticed her activity on Meetup and used it to trace her other online activities, eventually linking her to posts boasting about the data theft on Twitter and the Slack messaging service.  “I’ve basically strapped myself with a bomb vest, dropping capital ones dox and admitting it,” Thompson posted on Slack, prosecutors say. Highly sensitive financial and social insurance data compromised The stolen data was stored in Amazon S3, "An AWS spokesman confirmed that the company’s cloud had stored the Capital One data that was stolen, and said it wasn’t accessed through a breach or vulnerability in AWS systems", Bloomberg reports. Capital One said the largest category of information accessed was information on consumers and small businesses as of the time they applied for one of its credit card products from 2005 through early 2019. The breached data included personal information Capital One routinely collects at the time it receives credit card applications, including names, addresses, zip codes/postal codes, phone numbers, email addresses, dates of birth, and self-reported income. The hacker also obtained customer status data, e.g., credit scores, credit limits, balances, payment history, contact information including fragments of transaction data from a total of 23 days during 2016, 2017 and 2018. For the Canadian credit card customers, approximately 1 million Social Insurance Numbers were compromised in this incident. About 140,000 Social Security numbers of Capital One's credit card customers and about 80,000 linked bank account numbers of our secured credit card customers were compromised. Richard D. Fairbank, Capital One’s chief executive officer, said in a statement, "I am deeply sorry for what has happened. I sincerely apologize for the understandable worry this incident must be causing those affected.” Thompson is charged with computer fraud and faces a maximum penalty of five years in prison and a $250,000 fine. U.S. Magistrate Judge Mary Alice Theiler ordered Thompson to be held. A bail hearing is set for Aug 1. Capital One said, it “will notify affected individuals through a variety of channels. We will make free credit monitoring and identity protection available to everyone affected”. Capital One's justification of "Facts" is unsatisfactory Users are very skeptical about trusting Capital One with their data going ahead. A user on Hacker News writes, “Obviously this person committed a criminal act, however, Capital One should also shoulder responsibility for not securing customer data. I have a feeling we'd be waiting a long time for accountability on C1's part.” Security experts are surprised with Capital One’s stating of “facts that say “no Social Security numbers were breached’ and say this cannot be true. https://twitter.com/zackwhittaker/status/1156027826912428032 https://twitter.com/DavidAns/status/1156014432511643649 https://twitter.com/GossiTheDog/status/1156232048975273986 Similar to Capital One, there were other data breaches in the past where the companies have agreed on a settlement to help the affected customers like the Equifax or have been levied with huge fines like the Marriott International and British Airways. The Equifax data breach that affected 143 million U.S. consumers on September 7, 2017, resulted in a global settlement including up to $425 million to help people affected by the data breach amounting to approximately $125 per affected victim, should they apply for compensation. This global settlement was done with the Federal Trade Commission, the Consumer Financial Protection Bureau, and 50 U.S. states and territories. The Marriott data breach occurred in Marriott’s Starwood guest database that compromised 383 million user data was revealed on November 19, 2018. Recently, the Information Commissioner’s Office (ICO) in the UK announced its plans to impose a fine of more than £99 million ($124 million) under GDPR. The British Airways data breach compromised personal identification information of over 500,000 customers and is believed to have begun in June 2018. Early this month, the ICO also announced it will fine British Airways with more than £183m fine. As a major data breach in one of the largest banks, Capital One could feel the pinch by regulators soon. What sets this case apart from the above breaches is that the affected customers are from the US and Canada and not from the EU. In the absence of regulatory action by the ICO or the EU commission, it is yet to be seen if regulators in the US and Canada will rise to the challenge. Also, now that the alleged hacker has been arrested, does this mean Capital One could slip by without paying any significant fine? Only time can tell if Capital One will pay a huge sum to the regulators for not being watchful of their customers' data in two different states. If the Equifax-FTC case and the Facebook-FTC proceedings are any sign of things to come, Capital One has not much to be concerned about. To know more about this news in detail, read Capital One’s official announcement. Thompson faces additional charges for hacking into the AWS accounts of about 30 organizations On 28th August, an indictment was filed in a US federal district court, where the investigators mentioned they have identified most of the companies and institutions allegedly hit by Thompson. The prosecutors said Thompson wrote software that scanned for customer accounts hosted by a “cloud computing company,” which is believed to be her former employer, AWS or Amazon Web Services. "It is claimed she specifically looked for accounts that suffered a common security hole – specifically, a particular web application firewall misconfiguration – and exploited this weakness to hack into the AWS accounts of some 30 organizations, and siphon their data to her personal server. She also used the hacked cloud-hosted systems to mine cryptocurrency for herself, it is alleged," The Register reports. “The object of the scheme was to exploit the fact that certain customers of the cloud computing company had misconfigured web application firewalls on the servers that they rented or contracted from the cloud computing company,” the indictment reads. The indictment further reads, “The object was to use that misconfiguration in order to obtain credentials for accounts of those customers that had permission to view and copy data stored by the customers on their cloud computing company servers. The object then was to use those stolen credentials in order to access and copy other data stored by the customers.” Thus, she also faces a computer abuse charge over the 30 other AWS-hosted organizations she allegedly hacked and stole information from. Facebook fails to fend off a lawsuit over a data breach of nearly 30 million users US Customs and Border Protection reveal data breach that exposed thousands of traveler photos and license plate images Over 19 years of ANU(Australian National University) students’ and staff data breached

Last week Go programmers had a gala time learning, networking and programming at the Marriott Marquis San Diego Marina as the most awaited event GopherCon 2019 was held starting from 24th July till 27th July. GopherCon this year hit the road at San Diego with some exceptional conferences, and many exciting announcements for more than 1800 attendees from around the world. One of the attendees, Andrea Santillana Fernández, says the Go Community is growing, and doing quite well. She wrote on her blog post on the Source graph website that there are 1 million Go programmers around the world and month on month its membership keeps increasing. Indeed there is a significant growth in the Go community, so what did it have in store for the programmers at this year’s GopherCon 2019: On the road to Go 2 The major milestones for the journey to Go 2 were presented by Russ Coxx on Wednesday last week. He explained the main areas of focus for Go 2, which are as below: Error handling Russ notes that writing a program correctly without errors is hard. But writing a program correctly accounting for errors and external dependencies is much more difficult. He listed down a few errors which led in introducing error handling helpers like an optional Unwrap interface, errors.Is and errors.As in Go 1.13 version. Generics Russ spoke about Generics and said that they started exploring a new design since last year. They are working with programming language theory experts on the problem to help refine the proposal of generics code in Go. In a separate session, Ian Lance Taylor, introduced generics codes in Go. He briefly explained the need, implementation and benefits from generics for the Go language. Next, Taylor reviewed the Go contract design draft which included the addition of optional type parameters to types and functions. Taylor defined generics as “Generic programming which enables the representation of functions and data structures in a generic form, with types factored out.” Generic code is written using types, which are specified later. An unspecified type is called as type parameter. A type parameter offers support only when permitted by contracts. A generic code imparts strong basis for sharing codes and building programs. It can be compiled using an interface-based approach which optimizes time as the package is compiled only once. If a generic code is compiled multiple times, it can carry compile time cost. Ian showed a few sample codes written in Generics in Go. Dependency management In Go 2 the team wants to focus on Dependency management and explicitly refer to dependencies similar to Java. Russ explained this by giving a history of how in 2011 they introduced GOPATH to separate the distribution from the actual dependencies so that users could run multiple different distributions and to separate the concerns of the distribution from the external libraries. Then in 2015, they introduced the go vendor spec to formalize the vendor directory and simplify dependency management implementations. But in practice it did not work well. In 2016, they formed the dependency working group. This team started work on dep: a tool to reshape all the existing tools into one.The problem with dep and the vendor directory was multiple distinct incompatible versions of a dependency were represented by one import path. It is now called as the "Import Compatibility Rule". The team took what worked well and learned from VGo. VGo provides package uniqueness without breaking builds. VGo dictates different import paths for incompatible package versions. The team grouped similar packages and gave these groups a name: Modules. The VGo system is now go modules. It now integrates directly with the Go command. The challenge presented going forward is mostly around updating everything to use modules. Everything needs to be updated to work with the new conventions to work well. Tooling Finally, as a result of all these changes, they distilled and refined the Go toolchain. One of the examples of this is gopls or "Go Please". Gopls aims to create a smoother, standard interface to integrate with all editors, IDEs, continuous integration and others. Simple, portable and efficient graphical interfaces in Go Elias Naur presented Gio, a new open source Go library for writing immediate mode GUI programs that run on all the major platforms: Android, iOS/tvOS, macOS, Linux, Windows. The talk covered Gio's unusual design and how it achieves simplicity, portability and performance. Elias said, “I wanted to be able to write a GUI program in GO that I could implement only once and have it work on every platform. This, to me, is the most interesting feature of Gio.” https://twitter.com/rakyll/status/1154450455214190593 Elias also presented Scatter which is a Gio program for end-to-end encrypted messaging over email. Other features of Gio include: Immediate mode design UI state owned by program Only depends on lowest-level platform libraries Minimal dependency tree to keep things low level as possible GPU accelerated vector and text rendering It’s super efficient No garbage generated in drawing or layout code Cross platform (macOS, Linux, Windows, Android, iOS, tvOS, Webassembly) Core is 100% Go while OS-specific native interfaces are optional Gopls, new tool serves as a backend for Go editor Rebecca Stambler, mentioned in her presentation that the Go community has built many amazing tools to improve the Go developer experience. However, when a maintainer disappears or a new Go release wreaks havoc, the Go development experience becomes frustrating and complicated. To solve this issue, Rebecca revealed the details behind a new tool: gopls (pronounced as 'go please'). The tool is currently in development by the Go team and community, and it will ultimately serve as the backend for your Go editor. Below listed functionalities are expected from gopls: Show me errors, like unused variables or typos autocomplete would be nice function signature help, because we often forget While we're at it, hover-accessible "tooltip" documentation in general Help me jump to a variable that is needed to see An outline of package structure Get started with WebAssembly in Go WebAssembly in Go is here and ready to try! Although the landscape is evolving quickly, the opportunity is huge. The ability to deliver truly portable system binaries could potentially replace JavaScript in the browser. WebAssembly has the potential to finally realize the goal of being platform agnostic without having to rely on a JVM. In a session by Johan Brandhorst who introduces the technology, shows how to get started with WebAssembly and Go, discusses what is possible today and what will be possible tomorrow. As of Go 1.13, there is experimental support for WebAssembly using the JavaScript interface but as it is only experimental, using it in production is not recommended. Support for the WASI interface is not currently available but has been planned and may be available as early as in Go 1.14. Better x86 assembly generation from Go Michael McLoughlin in his presentation made the case for code generation techniques for writing x86 assembly from Go. Michael introduced assembly, assembly in Go, the use cases for when you would want to drop into assembly, and techniques for realizing speedups using assembly. He pointed out that most of the time, pure Go will be enough for 97% of programs, but there are those 3% of cases where it is warranted, and the examples he brought up were crypto, syscalls, and scientific computing. Michael then introduced a package called avo which makes high-performance Go assembly easier to write. He said that writing your assembly in Go will allow you to realize the benefits of a high level language such as code readability, the ability to create loops, variables, and functions, and parameterized code generation all while still realizing the benefits of writing assembly. Michael concluded the talk with his ideas for the future of avo. Use avo in projects specifically in large crypto implementations. More architecture support Possibly make avo an assembler itself (these kinds of techniques are used in JIT compilers) avo based libraries (avo/std/math/big, avo/std/crypto) The audience appreciated this talk on Twitter. https://twitter.com/darethas/status/1155336268076576768 The presentation slides for this are available on the blog. Miniature version of Golang, TinyGo for microcontrollers Ron Evans, creator of GoCV, GoBot and "technologist for hire" introduced TinyGo that can run directly on microcontrollers like Arduino and more. TinyGo uses the LLVM compiler toolchain to create native code that can run directly even on the smallest of computing devices. Ron demonstrated how Go code can be run on embedded systems using TinyGo, a compiler intended for use in microcontrollers, WebAssembly (WASM), and command-line tools. Evans began his presentation by countering the idea that Go, while fast, produces executables too large to run on the smallest computers. While that may be true of the standard Go compiler, TinyGo produces much smaller outputs. For example: "Hello World" program compiled using Go 1.12 => 1.1 MB Same program compiled using TinyGo 0.7.0 => 12 KB TinyGo currently lacks support for the full Go language and Go standard library. For example, TinyGo does not have support for the net package, although contributors have created implementations of interfaces that work with the WiFi chip built into Arduino chips. Support for Go Routines is also limited, although simple programs usually work. Evans demonstrated that despite some limitations, thanks to TinyGo, the Go language can still be run in embedded systems. Salvador Evans, son of Ron Evans, assisted him for this demonstration. At age 11, he has become the youngest GopherCon speaker so far. https://twitter.com/erikstmartin/status/1155223328329625600 There were talks by other speakers on topics like, improvements in VSCode for Golang, the first open source Golang interpreter with complete support of the language spec, Athens Project which is a proxy server in Go and how mobile development works in Go. https://twitter.com/ramyanexus/status/1155238591120805888 https://twitter.com/containous/status/1155191121938649091 https://twitter.com/hajimehoshi/status/1155184796386988035 Apart from these there were a whole lot of other talks which happened at the GopherCon 2019. There were live blogs posted by the attendees on various talks and till now more than 25 blogs are posted by the attendees on the Sourcegraph website. The Go team shares new proposals planned to be implemented in Go 1.13 and 1.14 Go introduces generic codes and a new contract draft design at GopherCon 2019 Is Golang truly community driven and does it really matter?  

Tõnis Tiigi, a software engineer at Docker and also a maintainer of Moby/Docker engine, in his recent post on Medium, explained how users can now leverage Docker’s non-root user privileges with Docker 19.03 release. He explains the Docker engine provides functionalities which are often tightly coupled to that of the Linux Kernel. For instance, to create namespaces in Linux users need privileged capabilities this is because a component of container isolation is based on Linux namespaces. “Historically Docker daemon has always needed to be started by the root user”, Tiigi explains. Docker is looking forward to changing this notion by introducing rootless support. With the help of Moby and BuildKit maintainer, Akihiro Suda, “we added rootless support to BuildKit image builder in 2018 and from February 2019 the same rootless support was merged to Moby upstream and is available for all Docker users to try in experimental mode”, Tiigi mentions. The rootless mode will help reduce the security footprint of the daemon and expose Docker capabilities to systems where users cannot gain root privileges. Rootless Docker and its benefits As the name suggests, a rootless mode in Docker allows a user to run Docker daemon, including the containers, as a non-root user on the host. The benefit to this is, even if it gets compromised the attacker will not be able to gain root access to the host. Akhiro Suda in his presentation, “Hardening Docker daemon with Rootless mode” explains rootless mode does not entirely fix vulnerabilities and misconfigurations but can mitigate attacks. With rootless mode attacker would not be able to access files owned by other users, modify firmware and kernel with an undetectable malware, or perform ARP spoofing. A few Caveats to the rootless Docker mode Docker engineers say the rootless mode cannot be considered a replacement for the complete suite of Docker engine features. Some limitation to the rootless mode include: cgroups resource controls, apparmor security profiles, checkpoint/restore, overlay networks etc. do not work on rootless mode. Exposing ports from containers currently requires manual socat helper process. Only Ubuntu-based distros support overlay filesystems in rootless mode. Rootless mode is currently only provided for nightly builds that may not be as stable as you are used to. As a lot of Linux features that Docker needs require privileged capabilities, the rootless mode takes advantage of user namespaces. “User namespaces map a range of user ID-s so that the root user in the inner namespace maps to an unprivileged range in the parent namespace. A fresh process in user namespace also picks up a full set of process capabilities”, Tiigi explains. Source: Medium In the recent release of the experimental rootless mode on GitHub, engineers mention rootless mode allows running dockerd as an unprivileged user, using user_namespaces(7), mount_namespaces(7), network_namespaces(7). Users need to run dockerd-rootless.sh instead of dockerd. $ dockerd-rootless.sh --experimental As Rootless mode is experimental, users need to always run dockerd-rootless.sh with --experimental. To know more about the Docker rootless mode in detail, read Tõnis Tiigi’s Medium post. You can also check out Akihiro Suda’s presentation Hardening Docker daemon with Rootless mode. Docker announces collaboration with Microsoft’s .NET at DockerCon 2019 Announcing Docker Enterprise 3.0 Public Beta! Are Debian and Docker slowly losing popularity?

Laracon US 2019, probably the biggest Laravel conference, wrapped up yesterday. Its creator, Tylor Otwell kick-started the event by talking about the next major release, Laravel 6. He also showcased a project that he has been working on called Laravel Vapor, a full-featured serverless management and deployment dashboard for PHP/Laravel. https://twitter.com/taylorotwell/status/1154168986180997125 This was a two-day event from July 24-25 hosted at Time Square, NYC. The event brings together people passionate about creating applications with Laravel, the open-source PHP web framework. Many exciting talks were hosted at this game-themed event. Evan You, the creator of Vue, was there presenting what’s coming in Vue.js 3.0. Caleb Porzio, a developer at Tighten Co., showcased a Laravel framework named Livewire that enables you to build dynamic user interfaces with vanilla PHP.  Keith Damiani, a Principal Programmer at Tighten, talked about graph database use cases. You can watch this highlights video compiled by Romega Digital to get a quick overview of the event: https://www.youtube.com/watch?v=si8fHDPYFCo&feature=youtu.be Laravel 6 coming next month Since its birth, Laravel has followed a custom versioning system. It has been on 5.x release version for the past four years now. The team has now decided to switch to semantic versioning system. The framework currently stands at version 5.8, and instead of calling the new release 5.9 the team has decided to go with Laravel 6, which is scheduled for the next month. Otwell emphasized that they have decided to make this change to bring more consistency in the ecosystem as all optional Laravel components such as Cashier, Dusk, Valet, Socialite use semantic versioning. This does not mean that there will be any “paradigm shift” and developers have to rewrite their stuff. “This does mean that any breaking change would necessitate a new version which means that the next version will be 7.0,” he added. With the new version comes new branding Laravel gets a fresh look with every major release ranging from updated logos to a redesigned website. Initially, this was a volunteer effort with different people pitching in to help give Laravel a fresh look. Now that Laravel has got some substantial backing, Otwell has worked with Focus Lab, one of the top digital design agencies in the US. They have together come up with a new logo and a brand new website. The website looks easy to navigate and also provides improved documentation to give developers a good reading experience. Source: Laravel Laravel Vapor, a robust serverless deployment platform for Laravel After giving a brief on version 6 and the updated branding, Otwell showcased his new project named Laravel Vapor. Currently, developers use Forge for provisioning and deploying their PHP applications on DigitalOcean, Linode, AWS, and more. It provides painless Virtual Private Server (VPS) management. It is best suited for medium and small projects and performs well with basic load balancing. However, it does lack a few features that could have been helpful for building bigger projects such as autoscaling. Also, developers have to worry about updating their operating systems and PHP versions. To address these limitations, Otwell created this deployment platform. Here are some of the advantages Laravel Vapor comes with: Better scalability: Otwell’s demo showed that it can handle over half a million requests with an average response time of 12 ms. Facilitates collaboration: Vapor is built around teams. You can create as many teams as you require by just paying for one single plan. Fine-grained control: It gives you fine-grained control over what each team member can do. You can set what all they can do across all the resources Vapor manages. A “vanity URL” for different environments: Vapor gives you a staging domain, which you can access with what Otwell calls a “vanity URL.” It enables you to immediately access your applications with “a nice domain that you can share with your coworkers until you are ready to assign a custom domain,“ says Otwell. Environment metrics: Vapor provides many environment metrics that give you an overview of an application environment. These metrics include how many HTTP requests have the application got in the last 24 hours, how many CLI invocations, what’s the average duration of those things, how much these cost on lambda, and more. Logs: You can review and search your recent logs right from the Vapor UI. It also auto-updates when any new entry comes in the log. Databases: With Vapor, you can provision two types of databases: fixed-sized database and serverless database. The fixed-sized database is the one where you have to pick its specifications like VCPU, RAM, etc. In the serverless one, however, if you do not select these specifications and it will automatically scale according to the demand. Caches: You can create Redis clusters right from the Vapor UI with as many nodes as you want. It supports the creation and management of elastic Redis cache clusters, which can be scaled without experiencing any downtime. You can attach them to any of the team’s projects and use them with multiple projects at the same time. To watch the entire demonstration by Otwell check out this video: https://www.youtube.com/watch?v=XsPeWjKAUt0&feature=youtu.be Laravel 5.7 released with support for email verification, improved console testing Building a Web Service with Laravel 5 Symfony leaves PHP-FIG, the framework interoperability group

Last year, Andrew Godwin, a Django contributor, formulated a roadmap to bring async functionality into Django. After a lot of discussion and amendments, the Django Technical Board approved his DEP 0009: Async-capable Django yesterday. Godwin wrote in a Google group, “After a long and involved vote, I can announce that the Technical Board has voted in favour of DEP 0009 (Async Django), and so the DEP has been moved to the "accepted" state.” The reason why Godwin thinks that this is the right time to bring async-native support in Django is that starting from version 2.1, it supports Python 3.5 and up. These Python versions have async def and similar native support for coroutines. Also, the web is now slowly shifting to use cases that prefer high concurrency workloads and large parallelizable queries. The motivation behind Async in Django The Django Enhancement Proposal (DEP) 0009 aims to address one of the core flaws in Python: inefficient threading. Python is not considered to be a perfect asynchronous language. Its ‘asyncio’ library for writing concurrent code suffers from some core design flaws. There are alternative async frameworks for Python but are incompatible. Django Channels brought some async support to Django but they primarily focus on WebSocket handling. Explaining the motivation, the DEP says, “At the same time, it's important we have a plan that delivers our users immediate benefits, rather than attempting to write a whole new Django-size framework that is natively asynchronous from the start.” Additionally, most developers are unacquainted with developing Python applications that have async support. There is also a lack of proper documentation, tutorials, and tooling to help them. Godwin believes that Django can become a “good catalyst” to help in creating guidance documentation. Goals this DEP outlines to achieve The DEP proposes to bring support for asynchronous Python into Django while maintaining synchronous Python support as well in a backward-compatible way. Here are its end goals, that Godwin listed in his roadmap: Making the blocking parts in Django such as sessions, auth, the ORM, and handlers asynchronous natively with a synchronous wrapper exposed on top where needed to ensure backward compatibility. Keeping familiar models/views/templates/middleware layout intact with very few changes. Ensuring that these updates do not compromise speed and cause significant performance regressions at any stage of this plan. Enabling developers to write fully-async websites if they want to, but not enforcing this as the default way of writing websites. Welcoming new talent into the Djang team to help out on large-scale features. Timeline to achieve these goals Godwin in his "A Django Async Roadmap" shared the following timeline: Django Version Updates 2.1 Current in-progress release. No async work 2.2 Initial work to add async ORM and view capability, but everything defaults to sync by default, and async support is mostly threadpool-based. 3.0 Rewrite the internal request handling stack to be entirely asynchronous, add async middleware, forms, caching, sessions, auth. Start the deprecation process for any APIs that are becoming async-only. 3.1 Continue improving async support, potential async templating changes 3.2 Finish deprecation process and have a mostly-async Django. Godwin posted a summary of the discussion he had with the Django Technical Board in the Google Group. Some of the queries they raised were how the team plans to distinguish async versions of functions/method from sync ones, how this implementation will ensure that there is no performance hit if the user opts out of async mode, and more. In addition to these technical queries, the board also raised a non-technical concern, “The Django project has lost many contributors over the years, is essentially in a maintenance mode, and we likely do not have the people to staff a project like this.” Godwin sees a massive opportunity to lurking in this fundamental challenge - namely to revive the Django project. He adds, “I agree with the observation that things have substantially slowed down, but I personally believe that a project like async is exactly what Django needs to get going again. There's now a large amount of fertile ground to change and update things that aren't just fixing five-year-old bugs.” Read the DEP 0009: Async-capable Django to know more in detail. Which Python framework is best for building RESTful APIs? Django or Flask? Django 2.2 is now out with classes for custom database constraints  

Microsoft has invested $1 billion in OpenAI with the goal of building next-generation supercomputers and a platform within Microsoft Azure which will scale to AGI (Artificial General Intelligence). This is a multiyear partnership with Microsoft becoming OpenAI’s preferred partner for commercializing new AI technologies. Open AI will become a big Azure customer, porting its services to run on Microsoft Azure. The $1 billion is a cash investment into OpenAI LP, which is Open AI’s for-profit corporate subsidiary. The investment will follow a standard capital commitment structure which means OpenAI can call for it, as they need it. But the company plans to spend it in less than five years. Per the official press release, “The companies will focus on building a computational platform in Azure for training and running advanced AI models, including hardware technologies that build on Microsoft’s supercomputing technology. These will be implemented in a safe, secure and trustworthy way and is a critical reason the companies chose to partner together.” They intend to license some of their pre-AGI technologies, with Microsoft becoming their preferred partner. “My goal in running OpenAI is to successfully create broadly beneficial A.G.I.,” Sam Altman, who co-founded Open AI with Elon Musk, said in a recent interview. “And this partnership is the most important milestone so far on that path.” Musk left the company in February 2019, to focus on Tesla and because he didn’t agree with some of what OpenAI team wanted to do. What does this partnership mean for Microsoft and Open AI OpenAI may benefit from this deal by keeping their innovations private which may help commercialization, raise more funds and get to AGI faster. For OpenAI this means the availability of resources for AGI, while potentially allowing founders and other investors with the opportunity to either double-down on OpenAI or reallocate resources to other initiatives However, this may also lead to them not disclosing progress, papers with details, and open source code as much as in the past. https://twitter.com/Pinboard/status/1153380118582054912 As for Microsoft, this deal is another attempt in quietly taking over open source. First, with the acquisition of GitHub and the subsequent launch of GitHub Sponsors, and now with becoming OpenAI’s ‘preferred partner’ for commercialization. Last year at an Investor conference, Nadella said, “AI is going to be one of the trends that is going to be the next big shift in technology. It's going to be AI at the edge, AI in the cloud, AI as part of SaaS applications, AI as part of in fact even infrastructure. And to me, to be the leader in it, it's not enough just to sort of have AI capability that we can exercise—you also need the ability to democratize it so that every business can truly benefit from it. That to me is our identity around AI.” Partnership with OpenAI seems to be a part of this plan. This deal can also possibly help Azure catch up with Google and Amazon both in hardware scalability and Artificial Intelligence offerings. A hacker news user comments, “OpenAI will adopt and make Azure their preferred platform. And Microsoft and Azure will jointly "develop new Azure AI supercomputing technologies", which I assume is advancing their FGPA-based deep learning offering. Google has a lead with TensorFlow + TPUs and this is a move to "buy their way in", which is a very Microsoft thing to do.” https://twitter.com/soumithchintala/status/1153308199610511360 It is also likely that Microsoft is investing money which will eventually be pumped back into its own company, as OpenAI buys computing power from the tech giant. Under the terms of the contract, Microsoft will eventually become the sole cloud computing provider for OpenAI, and most of that $1 billion will be spent on computing power, Altman says. OpenAI, who were previously into building ethical AI will now pivot to build cutting edge AI and move towards AGI. Sometimes even neglecting ethical ramifications, wanting to deploy tech at the earliest which is what Microsoft would be interested in monetizing. https://twitter.com/CadeMetz/status/1153291410994532352 I see two primary motivations: For OpenAI—to secure funding and to gain some control over hardware which in turn helps differentiate software. For MSFT—to elevate Azure in the minds of developers for AI training. - James Wang, Analyst at ARKInvest https://twitter.com/jwangARK/status/1153338174871154689 However, the news of this investment did not go down well with some experts in the field who saw this as a pure commercial deal and questioned whether OpenAI’s switch to for-profit research undermines its claims to be “democratizing” AI. https://twitter.com/fchollet/status/1153489165595504640 “I can't really parse its conversion into an LP—and Microsoft's huge investment—as anything but a victory for capital” - Robin Sloan, Author https://twitter.com/robinsloan/status/1153346647339876352 “What is OpenAI? I don't know anymore.” - Stephen Merity, Deep learning researcher https://twitter.com/Smerity/status/1153364705777311745 https://twitter.com/SamNazarius/status/1153290666413383682 People are also speculating whether creating AGI is really even possible. In a recent survey experts estimated that there was a 50 percent chance of creating AGI by the year 2099. Pet New York Times, most experts believe A.G.I. will not arrive for decades or even centuries. Even Altman admits OpenAI may never get there. But the race is on nonetheless. Then why is Microsoft delivering the $1 billion over five years considering that is neither enough money nor enough time to produce AGI. Although, OpenAI has certainly impressed the tech community with its AI innovations. In April, OpenAI’s new algorithm that is trained to play the complex strategy game, Dota 2, beat the world champion e-sports team OG at an event in San Francisco, winning the first two matches of the ‘best-of-three’ series. The competition included a human team of five professional Dota 2 players and AI team of five OpenAI bots. In February, they released a new AI model GPT-2, capable of generating coherent paragraphs of text without needing any task-specific training. However experts felt that the move signalled towards ‘closed AI’ and propagated the ‘fear of AI’ for its ability to write convincing fake news from just a few words. Github Sponsors: Could corporate strategy eat FOSS culture for dinner? Microsoft is seeking membership to Linux-distros mailing list for early access to security vulnerabilities OpenAI: Two new versions and the output dataset of GPT-2 out!

On Thursday, The Register reported that Laurie Voss, the co-founder and chief data officer of JavaScript package registry, NPM Inc left the company. Voss’s last day in office was 1st July while he officially announced the news on Thursday. Voss joined NPM in January 2014 and decided to leave the company in early May this year. NPM has faced its share of unrest in the company in the past few months. In the month of March  5 NPM employees were fired from the company in an unprofessional and unethical way. Later 3 of those employees were revealed to have been involved in unionization and filed complaints against NPM Inc with the National Labor Relations Board (NLRB).  Earlier this month NPM Inc at the third trial settled the labor claims brought by these three former staffers through the NLRB. Voss’ s resignation will be third in line after Rebecca Turner, former core contributor who resigned in March and Kat Marchan, former CLI and community architect who resigned from NPM early this month. Voss writes on his blog, “I joined npm in January of 2014 as co-founder, when it was just some ideals and a handful of servers that were down as often as they were up. In the following five and a half years Registry traffic has grown over 26,000%, and worldwide users from about 1 million back then to more than 11 million today. One of our goals when founding npm Inc. was to make it possible for the Registry to run forever, and I believe we have achieved that goal. While I am parting ways with npm, I look forward to seeing my friends and colleagues continue to grow and change the JavaScript ecosystem for the better.” Voss also told The Register that he supported unions, “As far as the labor dispute goes, I will say that I have always supported unions, I think they're great, and at no point in my time at NPM did anybody come to me proposing a union,” he said. “If they had, I would have been in favor of it. The whole thing was a total surprise to me.” The Register team spoke to one of the former staffers of NPM and they said employees tend not to talk to management in the fear of retaliation and Voss seemed uncomfortable to defend the company’s recent actions and felt powerless to affect change. In his post Voss is optimistic about NPM’s business areas, he says, “Our paid products, npm Orgs and npm Enterprise, have tens of thousands of happy users and the revenue from those sustains our core operations.” However, Business Insider reports that a recent NPM Inc funding round of the company raised only enough to continue operating until early 2020. https://twitter.com/coderbyheart/status/1152453087745007616 A big question on everyone’s mind currently is the stability of the public Node JS Registry. Most users in the JavaScript community do not have a fallback in place. While the community see Voss’s resignation with appreciation for his accomplishments, some are disappointed that he could not raise his voice against these odds and had to quit. "Nobody outside of the company, and not everyone within it, fully understands how much Laurie was the brains and the conscience of NPM," Jonathan Cowperthwait, former VP of marketing at NPM Inc, told The Register. CJ Silverio, a principal engineer at Eaze who served as NPM Inc's CTO said that it’s good that Voss is out but she wasn't sure whether his absence would matter much to the day-to-day operations of NPM Inc. Silverio was fired from NPM Inc late last year shortly after CEO Bryan Bogensberger’s arrival. “Bogensberger marginalized him almost immediately to get him out of the way, so the company itself probably won’t notice the departure," she said. "What should affect fundraising is the massive brain drain the company has experienced, with the entire CLI team now gone, and the registry team steadily departing. At some point they’ll have lost enough institutional knowledge quickly enough that even good new hires will struggle to figure out how to cope." Silverio also mentions that she had heard rumors of eliminating the public registry while only continuing with their paid enterprise service, which will be like killing their own competitive advantage. She says if the public registry disappears there are alternative projects like the one spearheaded by Silverio and a fellow developer Chris Dickinson, Entropic. Entropic is available under an open source Apache 2.0 license, Silverio says "You can depend on packages from any other Entropic instance, and your home instance will mirror all your dependencies for you so you remain self-sufficient." She added that the software will mirror any packages installed by a legacy package manager, which is to say npm. As a result, the more developers use Entropic, the less they'll need NPM Inc's platform to provide a list of available packages. Voss feels the scale of npm is 3x bigger than any other registry and boasts of an extremely fast growth rate i.e approx 8% month on month. "Creating a company to manage an open source commons creates some tensions and challenges is not a perfect solution, but it is better than any other solution I can think of, and none of the alternatives proposed have struck me as better or even close to equally good." he said. With  NPM Inc. sustainability at stake, the JavaScript community on Hacker News discussed alternatives in case the public registry comes to an end. One of the comments read, “If it's true that they want to kill the public registry, that means I may need to seriously investigate Entropic as an alternative. I almost feel like migrating away from the normal registry is an ethical issue now. What percentage of popular packages are available in Entropic? If someone else's repo is not in there, can I add it for them?” Another user responds, “The github registry may be another reasonable alternative... not to mention linking git hashes directly, but that has other issues.” Other than Entropic another alternative discussed is nixfromnpm, it is a tool in which you can translate NPM packages to Nix expression. nixfromnpm is developed by Allen Nelson and two other contributors from Chicago. Surprise NPM layoffs raise questions about the company culture Is the Npm 6.9.1 bug a symptom of the organization’s cultural problems? Npm Inc, after a third try, settles former employee claims, who were fired for being pro-union, The Register reports

A group of researchers from Intel Labs and Facebook have published a paper titled, “A Study of BFLOAT16 for Deep Learning Training”. The paper presents a comprehensive study indicating the success of Brain Floating Point (BFLOAT16) half-precision format in Deep Learning training across image classification, speech recognition, language modeling, generative networks and industrial recommendation systems. BFLOAT16 has a 7-bit mantissa and an 8-bit exponent, similar to FP32, but with less precision. BFLOAT16 was originally developed by Google and implemented in its third generation Tensor Processing Unit (TPU). https://twitter.com/JeffDean/status/1134524217762951168 Many state of the art training platforms use IEEE-754 or automatic mixed precision as their preferred numeric format for deep learning training. However, these formats lack in representing error gradients during back propagation. Thus, they are not able to satisfy the required  performance gains. BFLOAT16 exhibits a dynamic range which can be used to represent error gradients during back propagation. This enables easier migration of deep learning workloads to BFLOAT16 hardware. Image Source: BFLOAT16 In the above table, all the values are represented as trimmed full precision floating point values with 8 bits of mantissa with their dynamic range comparable to FP32. By adopting to BFLOAT16 numeric format, the core compute primitives such as Fused Multiply Add (FMA) can be built using 8-bit multipliers. This leads to significant reduction in area and power while preserving the full dynamic range of FP32. How Deep neural network(DNNs) is trained with BFLOAT16? The below figure shows the mixed precision data flow used to train deep neural networks using BFLOAT16 numeric format. Image Source: BFLOAT16 The BFLOAT16 tensors are taken as input to the core compute kernels represented as General Matrix Multiply (GEMM) operations. It is then forwarded to the FP32 tensors as output.   The researchers have developed a library called Quantlib, represented as Q in the figure, to implement the emulation in multiple deep learning frameworks. One of the functions of a Quantlib is to modify the elements of an input FP32 tensor to echo the behavior of BFLOAT16. Quantlib is also used to modify a copy of the FP32 weights to BFLOAT16 for the forward pass.   The non-GEMM computations include batch-normalization and activation functions. The  FP32 always maintains the bias tensors.The FP32 copy of the weights updates the step uses to maintain model accuracy. How does BFLOAT16 perform compared to FP32? Convolution Neural Networks Convolutional neural networks (CNN) are primarily used for computer vision applications such as image classification, object detection and semantic segmentation. AlexNet and ResNet-50 are used as the two representative models for the BFLOAT16 evaluation. AlexNet demonstrates that BFLOAT16 emulation follows very near to the actual FP32 run and achieves 57.2% top-1 and 80.1% top-5 accuracy. Whereas in ResNet-50, the BFLOAT16 emulation follows the FP32 baseline almost exactly and achieves the same top-1 and top-5 accuracy. Image Source: BFLOAT16 Similarly, the researchers were able to successfully demonstrate that BFLOAT16 is able to represent tensor values across many application domains including Recurrent Neural Networks, Generative Adversarial Networks (GANs) and Industrial Scale Recommendation System. The researchers thus established that the dynamic range of BFLOAT16 is of the same range as that of FP32 and its conversion to/from FP32 is also easy. It is important to maintain the same range as FP32 since no hyper-parameter tuning is required for convergence in FP32. A hyperparameter is a parameter of choosing a set of optimal hyperparameters in machine learning for a learning algorithm. Researchers of this paper expect to see an industry-wide adoption of BFLOAT16 across emerging domains. Recent reports suggest that Intel is planning to graft Google’s BFLOAT16 onto its processors  as well as on its initial Nervana Neural Network Processor for training, the NNP-T 1000. Pradeep Dubey, who directs the Parallel Computing Lab at Intel and is also one of the researchers of this paper believes that for deep learning, the range of the processor is more important than the precision, which is the inverse of the rationale used for IEEE’s floating point formats. Users are finding it interesting that a BFLOAT16 half-precision format is suitable for deep learning applications. https://twitter.com/kevlindev/status/1152984689268781056 https://twitter.com/IAmMattGreen/status/1152769690621448192 For more details, head over to the “A Study of BFLOAT16 for Deep Learning Training” paper. Intel’s new brain inspired neuromorphic AI chip contains 8 million neurons, processes data 1K times faster Google plans to remove XSS Auditor used for detecting XSS vulnerabilities from its Chrome web browser IntelliJ IDEA 2019.2 Beta 2 released with new Services tool window and profiling tools

The tech industry is being plagued by moral and ethical issues as top players are increasingly becoming explicit about prioritizing profits over people or planet. Recent times are rift with cases of tech companies actively selling facial recognition technology to law enforcement agencies, helping ICE separate immigrant families, taking large contracts with the Department of Defense, accelerating the extraction of fossil fuels, deployment of surveillance technology. As the US gets alarmingly dangerous for minority groups, asylum seekers and other vulnerable communities, it has awakened the tech worker community to organize for keeping their employers in check. They have been grouping together to push back against ethically questionable decisions made by their employers using the hashtag #TechWontBuildIt since 2018. Most recently, several open source communities, activists and developers have strongly demonstrated against Palantir for their involvement with ICE. Palantir, a data analytics company, founded by Peter Thiel, one of President Trump’s most vocal supporters in Silicon Valley, has been called out for its association with the Immigration and Customs Enforcement (ICE). According to emails obtained by WNYC, Palantir’s mobile app FALCON is being used by ICE to carry out raids on immigrant communities as well as enable workplace raids. According to the emails, an ICE supervisor sent an email to his officers before a planned spate of raids in New York City in 2017. The emails ordered them to use a Palantir program, called FALCON mobile, for the operation. The email was sent in preparation for a worksite enforcement briefing on January 8, 2018. Two days later, ICE raided nearly a hundred 7-Elevens across U.S. According to WNYC, ICE workplace raids led to 1,525 arrests over immigration status from October 2017 to October 2018. The email reads, “[REDACTION] we want all the team leaders to utilize the FALCON mobile app on your GOV iPhones, We will be using the FALCON mobile app to share info with the command center about the subjects encountered in the stores as well as team locations." Other emails obtained by WYNC detail a Palantir staffer notifying an ICE agent to test out their FALCON mobile application because of his or her “possible involvement in an upcoming operation.” Another message, in April 2017, shows a Palantir support representative instructing an agent on how to classify a datapoint, so that Palantir’s Investigative Case Management [ICM] platform could properly ingest records of a cell phone seizure. In December 2018, Palantir told the New York Times‘ Dealbook that Palantir technology is not used by the division of ICE responsible for carrying out the deportation and detention of undocumented immigrants. Palantir declined WNYC’s requests for comment. Citing law enforcement “sensitivities,” ICE also declined to comment on how it uses Palantir during worksite enforcement operations. In May this year, new documents released by Mijente, an advocacy organization, revealed that Palantir was responsible for 2017 operation that targeted and arrested family members of children crossing the border alone. The documents show a huge contrast to what Palantir said its software was doing. As part of the operation, ICE arrested 443 people solely for being undocumented. Mijente has then urged Palantir to drop its contract with ICE and stop providing software to agencies that aid in tracking, detaining, and deporting migrants, refugees, and asylum seekers. Open source communities, activists and developers strongly oppose Palantir Post the revelation of Palantir’s involvement with ICE, several open-source developers are strongly opposing Palantir. The Entropic project, a JS package registry, is debating the idea of banning Palantir employees from participating in the project. Kat Marchán, Entropic maintainer posted on the forum, “I find it unconscionable for tech folks to be building the technological foundations for this deeply unethical and immoral (and fascist) practice, and I would like it if we, in our limited power as a community to actually affect the situation, officially banned any Palantir employees from participating in or receiving any sort of direct support from the Entropic community.” She has further proposed explicitly banning Palantir employees from the Discourse, the Discord, as well as the GitHub communities and any other forums, Entropic may use for coordinating the project. https://twitter.com/maybekatz/status/1151355320314187776 Amazon is also facing renewed calls from employees and external immigration advocates to stop working with Palantir. According to an internal email obtained by Forbes, Amazon employees are recirculating a June 2018 letter to executives calling for Palantir to be kicked off Amazon Web Services. More than 500 Amazon employees have signed the letter addressed to CEO Jeff Bezos and AWS head Andy Jassy. Not just that, pro-immigration organizations such as Mijente and Jews for Racial and Economic Justice, interrupted the keynote speech at Amazon’s annual AWS Summit, last Thursday. https://twitter.com/altochulo/status/1149326296092164097 More than a dozen groups of activists also protested on July 12 against Palantir Technologies in Palo Alto for the company’s provision of software facilitating ICE raids, detentions, and deportations. City residents also joined the protests expanding the total to hundreds. Back in August 2018, the Lerna team had taken a strong stand against ICE by modifying their MIT license to ban companies who have collaborated with ICE from using Lerna. The updated license banned companies that are known collaborators with ICE such as Microsoft, Palantir, and Amazon, among the others from using Lerna. To quote Meredith Whittaker, Google walkout organizer who recently left the company, from her farewell letter, “Tech workers have emerged as a force capable of making real change, pushing for public accountability, oversight, and meaningful equity. And this right when the world needs it most” She further adds, “The stakes are extremely high. The use of AI for social control and oppression is already emerging, even in the face of developers’ best of intentions. We have a short window in which to act, to build in real guardrails for these systems before AI is built into our infrastructure and it’s too late.” Extraordinary times call for extraordinary measures. As the tech industry grapples with the consequences of its hypergrowth technosolutionist mindset, where do tech workers draw the line? Can tech workers afford to be apolitical or separate their values from the work they do? There are no simple answers, but one thing is for sure - the questions must be asked and faced. Open source, as part of the commons, has a key role to play and how it evolves in the next couple of years is likely to define the direction the world would take. Lerna relicenses to ban major tech giants like Amazon, Microsoft, Palantir from using its software as a protest against ICE Palantir’s software was used to separate families in a 2017 operation reveals Mijente ACLU files lawsuit against 11 federal criminal and immigration enforcement agencies for disclosure of information on government hacking.

A Qlik Sense application combines data in the form of a structured data model, visualizations, sheets, and stories. Dimensions, measures, and visualizations can be created as data items that can be reused between several visualizations (charts) and sheets (visualizations). This article is taken from the book Hands-On Business Intelligence with Qlik Sense by Kaushik Solanki, Pablo Labbe, Clever Anjos, and Jerry DiMaso. By the end of this book, you will be well-equipped to run successful business intelligence applications using Qlik Sense's functionality, data modeling techniques, and visualization best practices. To follow along with the examples implemented in this article, you can download the code from the book’s GitHub repository. In this article, we will create a sales analysis application to explore and analyze the data model that you can find on GitHub. While developing the application, we will apply the Dashboard, Analysis, Reporting (DAR) methodology. Technical requirements We will use the application we previously created, as a starting point, with a loaded data model to eliminate the process of loading and modeling the data all over again. You can also download the initial and final versions of the application from the book repository on GitHub. After downloading the initial version of the application, follow these steps: If you are using Qlik Sense Desktop, place the QVF application file in the QlikSense\Apps folder, under your document's personal folder. If you are using Qlik Sense Cloud, upload the application to your personal workspace. The problem statement Suppose that the company is a worldwide wholesale seller of food and beverages. This means that they need to review information about their sales in several countries. They need to know their performance in each of them, and they require this information to be detailed by the customer name, category name, and product name. They also need to know the average percentage of discounts, how many orders they have issued, and the total sales amount. Each of these key indicators should be dictated by the month and year. That explains the basis for this case. We need to find the perfect solution by accessing the right dashboard and displaying all of the data in the most consolidated fashion. Creating the dashboard sheet We will begin by creating a new sheet with the name Dashboard: Open the app and click on Create new sheet: Set the Title of the sheet to Dashboard: Click on the sheet icon to save the title, and open the sheet to start creating visualizations. Creating KPI visualizations A KPI visualization is used to get an overview of the performance values that are important to our company. To add the KPI visualizations to the sheet, follow these steps: Click on the Edit button located on the toolbar to enter the edit mode: Click on the Master items button on the asset panel and click on the Measures heading: Click on Sales $ and drag and drop it into the empty space on the sheet: Qlik Sense will create a new visualization of the KPI type because we have selected a measure: Resize the visualization toward the top-left of the sheet: Repeat steps 1 through 5 to add two visualizations for the Avg Discount % and Orders # measures. Place the objects to the right of the previously added visualization: To change the type of visualization from Gauge to KPI, click on the chart type selector: Select the KPI chart type: Now, all three of the measures are visualized as KPI: Creating a pie chart with Sales $ by Categories To add the pie chart with Sales $ by Categories onto the sheet, follow these steps: Click on the Charts button on the asset panel, which is on the left-hand side of the screen, to open the chart selector panel. Click on Pie chart and drag and drop it into the empty space on the sheet: Click on the Add dimension button and select Category in the Dimensions section: Click on the Add measure button and select Sales $ in the Measures section: The pie chart will look like this: Now, we will enhance the presentation of the chart by removing the Dimension label and adding a title to the chart: To remove the Dimension label, select the Appearance button that lies in the properties panel at the right-hand side of the screen and expand Presentation, under which you will find the Dimension label. Turn it off by simply clicking on the toggle button: Click on the title of the object and type Sales $ share by Category: Click on Done in the toolbar to enter the visualization mode: Creating a bar chart with Sales $ by Top 10 Customers To add the bar chart with the top 10 customers by sales $ to the sheet, carry out these steps: Before adding the bar chart, resize the pie chart: Click on the Charts button that lies on the asset panel to open the chart selector panel. Click on Bar chart and drag and drop it into the empty space in the center of the sheet: Click on the Add dimension button and select the Customer option in the Dimensions section. Click on the Add measure button and select Sales $ in the Measures section. The bar chart will look like this: To enhance the presentation of the chart, we will limit the number of customers that are depicted in the chart to 10, and add a title to the chart: Select Data in the properties panel on the right-hand side of the screen and expand the Customer dimension. Set the Limitation values as Fixed number, Top and type 11 in the limitation box: Click on the title of the chart and type Top 10 Customers by Sales $. Click on Done to enter the visualization mode. The bar chart will look like this: Creating the geographical map of sales by country To add the geographical map of sales by country to the sheet, follow these steps: Before adding the map chart, resize the bar chart: Click on the Charts button that lies on the asset panel to open the chart selector panel. Click on the Map button and drag and drop the chart into the empty space on the right-hand side of the sheet: The map visualization will show a default world map with no data, as follows: Here, we need to add an Area layer to plot the countries, and add a Sales $ measure to fill in the area of each country with a color scale: Click on the Add Layer button in the properties panel on the right-hand side of the screen: Select the Area layer: Add the Country dimension, as it contains the information to plot the area: The map will show the country areas filled in with a single color, as follows: To add the Sales $ measure to set the color scale for each country, go to the asset panel at the left-hand side of the screen and click on the Master items heading in the Measures section. Drag and drop the Sales $ measure on top of the map: In the pop-up menu for the map, select Use in "Country"(Area Layer): After that, select Color by: Sales $: The map will now show the countries with more Sales $ in a dark color, and those with lower Sales $ in a light color: Now, click on the title of the object and type Sales $ by Country. Click on the Done button to enter the visualization mode. The sheet will look like this, but it will vary according to your screen resolution: Creating the analysis sheet While the dashboard sheet shows information on several topics for a quick overview, the analysis sheet focuses on a single topic for data exploration. We will create the analysis sheet with the following visualizations: A filter panel, with the dimensions: OrderYear, OrderMonth, Country, Customer, Category, and Product KPI Sales $ KPI Avg Discount % A combo chart for Pareto (80/20) analysis by customer A table with customer data Let's start with creating a new sheet with the name Customer Analysis: Click on the Sheet selection button at the top-right of the screen to open the sheet overview panel. Click on the Create new sheet button and set the title of the sheet to Customer Analysis. To finish this example, click on the sheet icon to save the title, and open the sheet to start creating visualizations. Adding a filter pane with main dimensions We will now build the customer analysis sheet by adding a filter pane by following these steps: Click on the Edit button to enter the edit mode. Click on the Charts button on the asset panel and drag and drop Filter pane into the empty space on the sheet: Click on the Add dimension button and select Order Year in the Dimensions section: Since we need to add more dimensions to our Filter pane, click on the Add dimension button in the properties on the right-hand side of the screen, and select Order Month in the Dimensions section. Repeat the previous step to add the Country, Customer, Category, and Product dimensions. The Filter pane will look like what's shown in the following screenshot: Now, resize the width of the filter panel to fit three columns of the grid: We also need to add the Filter pane as a master visualization, which is to be reused across the analysis and reporting sheets that we will create next: Right-click on the filter pane and select Add to master items: Set the name of the master item to Default Filter and the description to A filter pane to be reused across sheets: Click on the Add button: Adding KPI visualizations To add the KPIs of Sales $ and Avg Discount % to the sheet, we have two options. The first option is to add the KPI visualizations to the Master items library, and add them to the new sheet: Go to the dashboard sheet. Add the KPI visualizations of Sales $ and Avg Discount % to the Master item. Name them KPI Sales $ and KPI Avg Discount %, respectively. From the visualization section in the Master items library, simply drag and drop each of the KPIs into the top end of the sheet. The second option is to copy and paste the KPI visualizations between sheets: Go to the dashboard sheet. Select the KPI visualization for Sales $. Press Ctrl + C or right-click on the visualization object and select Copy in the context menu. Go back to the Customer Analysis sheet. Press Ctrl + V or right-click in the empty area of the sheet and select Paste in the context menu. Repeat the same steps for KPI Avg Discount %. The sheet editor will look like this: Creating a combo chart for Pareto (80/20) analysis A Pareto analysis helps us to identify which groups of customers contribute to the first 80% of our sales. To create a Pareto analysis, we will use a combo chart as it allows us to combine metrics with different shapes such as bars, lines, and symbols. We will represent the data in two axes; the primary axis is found at the left-hand side of the chart, and the secondary axis is found at the right-hand side of the chart. In our example, the chart has a bar for Sales $ in the primary axis, as well as two lines: one for the Cumulative % of sales, and the other as static, with 80% in the secondary axis. In the following screenshot, you can see the highlighted customers contributing to the first 80% of the sales: To create the Pareto analysis chart, follow these steps: Click on the Charts button on the asset panel and find the Combo chart. Drag and drop the Combo chart into the empty space at the right-hand side of the sheet. Click on Add Dimension and select Customer in the Dimension section. Click on Add Measure and select Sales $ in the Measures section. The combo chart will look like this: We need to add two other measures, represented by lines. The first is the cumulative percentage of sales, and the second is the reference line at 80%. To add the cumulative sales line, go to the properties panel, expand the Data section, and click on the Add button in Measures: Click on the fx button to open the expression editor: Type the following expression in the expression editor to calculate a cumulative ratio of the sales for each customer, over the whole amount of the sales of all customers: RangeSum(Above(Sum(SalesAmount), 0, RowNo())) / Sum(total SalesAmount) Click on the Apply button to close the expression editor and save the expression. Set the Label of the new measure to Cumulative Sales %. Check if the properties Line and Secondary axis are selected for the measure: Change the number formatting to Number, set the formatting option to Simple, and select 12.3%. Now, find the Add button in the Measure pane to add another measure: the reference line for 80%. Open the Expression editor, type 0.8, and click on the Apply button. Set the Label to 80%. Check if the properties Line is selected and that the Secondary axis is selected for the measure: We also need to fix the sort order into a descending fashion, by Sales $: Go to the properties panel and expand the Sorting section. Click on Customer to expand the Sorting configuration for the dimension. Switch off the Auto sorting. Click on the checkbox for Sort by expression to select the option. Open the Expression editor and type sum(SalesAmount). Click on Apply to close the expression editor and apply the changes. Set the Title of the chart to Pareto Analysis. Change the sorting order to Descending. Deselect other sorting options if they are selected. The Sorting pane will look like this: Finally, the combo chart will look like this: Creating a reporting sheet Reporting sheets allow the user to see the data in a more granular form. This type of sheet provides information that allows the user to take action at an operational level. We will start this example by creating a new sheet with the name Reporting: Click on the Sheet selection button at the top-right of the screen to open the sheet overview panel Click on the Create new sheet button and set the Title of the sheet to Product Analysis Click on the sheet icon to save the title, open the sheet to start creating visualizations, and enter the edit mode Adding a default filter pane We will start to build the reporting sheet by adding the default filter pane that has already been added to the Master items library: Click on the Edit button to enter the edit mode. Click on the Master items button on the asset panel and find Default filter in the Visualization section. Click on Default filter pane and drag and drop it into the empty space at the top of the sheet. Resize the height of the filter pane to fit one row of the grid. The sheet will then look like this: Next, we will add the table chart to the sheet, as follows: Click on the Charts button on the asset panel and find the Table visualization. Click on Table and drag and drop it into the empty space at the center of the sheet. Click on the Add dimension button and select OrderID from the Field list. Click on Add measure and select Sales $ from the Dimensions list. Click on the Master items button on the asset panel, which is on the left-hand side of the screen, and click the Dimensions heading to expand it. We will then add more dimensions. Drag and drop the Customer dimension on the table. Select Add "Customer" from the floating menu. Repeat the process, using the drag and drop feature to add Country, Category, Product, EmployeesFirstName to the table. Click on the Measures heading in Master items to expand it. Drag and drop the Avg Discount % and Quantity # measures onto the table. Select Add  in the floating menu for each of the selected measure. Click on the Fields button on the asset panel, which is on the left-hand side of the screen. Find the OrderID field in the list. Drag and drop the OrderID field onto the table. Select Add OrderID from the floating menu. Repeat the same steps to add the OrderDate field to the table. The table will look like this: In this article, we saw how to create a Qlik Sense application using the DAR methodology, which will help you to explore and analyze an application's information. If you found this post useful, do check out the book, Hands-On Business Intelligence with Qlik Sense. This book teaches you how to create dynamic dashboards to bring interactive data visualization to your enterprise using Qlik Sense. Best practices for deploying self-service BI with Qlik Sense Four self-service business intelligence user types in Qlik Sense How Qlik Sense is driving self-service Business Intelligence

Yesterday the team at Nesta organization, an innovation firm based out of UK published a research on gender diversity in the AI research workforce. The authors of this research are Juan Mateos Garcis, the Director, Konstantinos Stathoulopoulos, the Principal Researcher and Hannah Owen, the Programme Coordinator at Nesta. https://twitter.com/JMateosGarcia/status/1151517641103872006 They have prepared an analysis purely based on 1.5 million arxiv papers. The team claims that it is the first ever study of gender diversity in AI which is not on any convenience sampling or proprietary database. The team posted on its official blog post, “We conducted a large-scale analysis of gender diversity in AI research using publications from arXiv, a repository with more than 1.5 million preprints widely used by the AI community. We aim to expand the evidence base on gender diversity in AI research and create a baseline with which to interrogate the impact of current and future policies and interventions.  To achieve this, we enriched the ArXiv data with geographical, discipline and gender information in order to study the evolution of gender diversity in various disciplines, countries and institutions as well as examine the semantic differences between AI papers with and without female co-authors.” With this research the team also aims to bring prominent female figures they have identified under the spotlight. Key findings from the research Serious gender diversity crisis in AI research The team found a severe gender diversity gap in AI research with only 13.83% of authors being women. Moreover, in relative terms, the proportion of AI papers co-authored by at least one woman has not improved since the 1990s. Juan Mateos thinks this kind of crisis is a waste of talent and it increases the risk of discriminatory AI systems. https://twitter.com/JMateosGarcia/status/1151517642236276736 Location and research domain are significant drivers of gender diversity Women in the Netherlands, Norway and Denmark are more likely to publish AI papers while those in Japan and Singapore are less likely. In the UK, 26.62% of the AI papers have at least one female co-author, placing the country at the 22nd spot worldwide. The US follows the UK in terms of having at least one female co-authors at 25% and for the unique female author US leads one position above UK. Source: Nesta research report Regarding the research domains, women working in Physics and Education, Computer Ethics and other societal issues and Biology are more likely to publish their work on AI in comparison to those working in Computer Science or Mathematics. Source: Nesta research report Significant gender diversity gap in universities, big tech companies and other research institutions Apart from the University of Washington, every other academic institution and organisation in the dataset has less than 25% female AI researchers. Regarding some of the big tech, only 11.3% of Google’s employees who have published their AI research on arXiv are women, while the proportion is similar for Microsoft (11.95%) and is slightly better for IBM (15.66%). Important semantic differences between AI paper with and without a female co-author When examining the publications in the Machine Learning and Societal topics in the UK in 2012 and 2015, papers involving at least one female co-author tend to be more semantically similar to each other than with those without any female authors. Moreover, papers with at least one female co-author tend to be more applied and socially aware, with terms such as fairness, human mobility, mental, health, gender and personality being among the most salient ones. Juan Mateos noted that this is an area which deserves further research. https://twitter.com/JMateosGarcia/status/1151517647361781760   The top 15 women with the most AI publications on arXiv identified Aarti Singh, Associate Professor at the Machine learning department of Carnegie Mellon University Cordelia Schmid, is a part of Google AI team and holds a permanent research position at Inria Grenoble Rhone-Alpes Cynthia Rudin, an associate professor of computer science, electrical and computer engineering, statistical science and mathematics at Duke University Devi Parikh, an Assistant Professor in the School of Interactive Computing at Georgia Tech Karen Livescu, an Associate Professor at Toyota Technical Institute at Chicago Kate Saenko,  an Associate Professor at the Department of Computer at Boston University Kristina Lerman, a Project Leader at the Information Sciences Institute at the University of Southern California Marilyn A. Walker, a Professor at the Department of Computer Science at the University of California Mihaela van der Schaar, is John Humphrey Plummer Professor of Machine Learning, Artificial Intelligence and Medicine at the University of Cambridge and a Turing Fellow at The Alan Turing Institute in London Petia Radeva, a professor at the Department of Mathematics and Computer Science, Faculty of Mathematics and Computer Science at the Universitat de Barcelona Regina Barzilay is a professor at the Massachusetts Institute of Technology and a member of the MIT Computer Science and Artificial Intelligence Laboratory Svetha Venkatesh, an ARC Australian Laureate Fellow, Alfred Deakin Professor and Director of the Centre for Pattern Recognition and Data Analytics (PRaDA) at Deakin University Xiaodan Liang, an Associate Professor at the School of Intelligent Systems Engineering, Sun Yat-sen University Yonina C. Elda, a Professor of Electrical Engineering, Weizmann Faculty of Mathematics and Computer Science at the University of Israel Zeynep Akata, an Assistant Professor with the University of Amsterdam in the Netherlands There are 5 other women researchers who were not identified in the study. Interviews bites from few women contributors and institutions The research team also interviewed few researchers and institutions identified in their work and they think a system wide reform is needed. When the team discussed the findings with the most cited female researcher Mihaela Van Der Schaar, she did feel that her presence in the field has only started to be recognised, having begun her career in 2003, ‘I think that part of the reason for this is because I am a woman, and the experience of (the few) other women in AI in the same period has been similar.’ she says. Professor Van Der Schaar also described herself and many of her female colleagues as ‘faceless’, she suggested that the work of celebrating leading women in the field could have a positive impact on the representation of women, as well as the disparity in the recognition that these women receive. This suggests that work is needed across the pipeline, not just with early-stage invention in education, but support for those women in the field. She also highlighted the importance of open discussion about the challenges women face in the AI sector and that workplace changes such as flexible hours are needed to enable researchers to participate in a fast-paced sector without sacrificing their family life. The team further discussed the findings with the University of Washington’s Eve Riskin, Associate Dean of Diversity and Access in the College of Engineering. Riskin described that much of her female faculty experienced a ‘toxic environment’ and pervasive imposter syndrome. She also emphasized the fact that more research is needed in terms of the career trajectories of the male and female researchers including the recruitment and retention. Some recent examples of exceptional women in AI research and their contribution While these women talk about the diversity gaps in this field recently we have seen works from female researchers like Katie Bouman which gained significant attention. Katie is a post-doctoral fellow at MIT whose algorithm led to an image of a supermassive black hole. But then all the attention became a catalyst for a sexist backlash on social media and YouTube. It set off “what can only be described as a sexist scavenger hunt,” as The Verge described it, in which an apparently small group of vociferous men questioned Bouman’s role in the project. “People began going over her work to see how much she’d really contributed to the project that skyrocketed her to unasked-for fame.” Another incredible example in the field of AI research and ethics is of Meredith Whittaker, an ex-Googler, now a program manager, activist, and co-founder of the AI Now Institute at New York University. Meredith is committed to the AI Now Institute, her AI ethics work, and to organize an accountable tech industry. On Tuesday,  Meredith left Google after facing retaliation from company for organizing last year’s protest of Google Walkout for Real Change demanding the company for structural changes to ensure a safe and conducive work environment for everyone.. Other observations from the research and next steps The research also highlights the fact that women are as capable as men in contributing to technical topics while they tend to contribute more than men to publications with a societal or ethical output. Some of the leading AI researchers in the field shared their opinion on this: Petia Radeva, Professor at the Department of Mathematics and Computer Science at the University of Barcelona, was positive that the increasingly broad domains of application for AI and the potential impact of this technology will attract more women into the sector. Similarly, Van Der Schaar suggests that “publicising the interdisciplinary scope of possibilities and career paths that studying AI can lead to will help to inspire a more diverse group of people to pursue it. In parallel, the industry will benefit from a pipeline of people who are motivated by combining a variety of ideas and applying them across domains.” The research team in future will explore the temporal co-authorship network of AI papers to examine how different the career trajectory of male and female researchers might be. They will survey AI researchers on arXiv and investigate the drivers of the diversity gap in more detail through their innovation mapping methods. They also plan to extend this analysis to identify the representation of other underrepresented groups. Meredith Whittaker, Google Walkout organizer, and AI ethics researcher is leaving the company, adding to its brain-drain woes over ethical concerns “I’m concerned about Libra’s model for decentralization”, says co-founder of Chainspace, Facebook’s blockchain acquisition DeepMind’s Alphastar AI agent will soon anonymously play with European StarCraft II players