Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-tidelift-is-building-a-new-model-for-open-source-software-airbnb-for-maintainers
Richard Gall
02 May 2019
6 min read
Save for later

Tidelift is building a new model for open source software: Airbnb for maintainers

Richard Gall
02 May 2019
6 min read
The challenges and issues within open source are well-documented. Indeed, some, such as Nadia Eghbal have suggested it's a timebomb waiting to explode. However, Tidelift is building what it believes is a better model for open source software. An announcement made at the end of April signalled a new chapter in Tidelift's young life - having secured $25 million Series B funding back in January, it has now extended its coverage of open source projects, with full 'support' for more than 1,000 of them, and new tools for subscribers. How does Tidelift support open source projects? At this point, you're probably wondering how Tidelift actually works: what does 'support' actually mean in the context of open source? Essentially, Tidelift is a bit like Uber or Airbnb for open source projects. The organization sits between the organizations that use open source software and those developers that maintain the software. Tidelift subscribers get open source software that is managed and maintained by the maintainers who, in turn, receive compensation from Tidelift. Co-founder Donald Fischer, who was previously an executive at Red Hat, explains the idea in a little more detail. "We were observing some of the business model developments in other contexts, specifically around this sort of managed market model, or gig economy model... and we started thinking hey that would apply pretty well to what's already happening with open source," he tells me over Skype. The problem, Fischer points out, is that "when you just download some raw open source project, you don't get a lot of the things that big companies expect from their software providers, like somebody being on the hook to keep it patched and secure, and double checking the licensing and so on..." Essentially, Tidelift bridges the gap between organizations that use open source software (which is, after all, basically all of them) and the people that maintain and up date it voluntarily. How does Tidelift actually work? On a practical level, one way of thinking about it is that Tidelift provides an SLA and insurance for engineering teams that use open source software. So, rather than simply downloading and using the "raw technology" in the way most of the engineering world has been over the last decade. So, when an engineering team purchases a Tidelift subscription, the open source software they use - provided it is working in partnership with Tidelift - is 'covered' in terms of standards, security, and licensing. It also provides technical support too, with the open source maintainers testing packages before they end up in a team's software. "We connect to your software development process through our software as a service tool." Fischer explains It sort of connects in a similar way to how a continuous integration testing tool connects to a GitHub or BitBucket... and every time your application code changes we look at what has changed as far as the open source packages [have changed]... and basically make sure all of that stuff meets whatever policy you've established with Tidelift." How does Tidelift differ from enterprise open source companies like Red Hat? Tidelift may look a little bit like a company like Red Hat. Indeed, Fischer's background at the company means some of the DNA of Red Hat has found its way into Tidelift. But despite both organizations making open source software more 'usable' for the enterprise, they are, in fact, very different indeed. The key issue for a company like Red Hat is dealing with the scale of open source tools available. "Red Hat hires full time employees and assigns them to shadow upstream open source products that are... typically being created by other third parties," Fischer explains. "There's a scale challenge there... you have to have expertise in each one of the packages that you're covering... The way we make it scale to this much broader universe of packages is that we go directly to the people that typically wrote the package in the first place." Fischer continues "we invite them to do the work that a Red Hat employee would do in the Linux context - and we give them the opportunity to get paid for that, which is typically pretty novel for them." The key challenges with open source that Tidelift is trying to solve Payment is just one of the issues facing the open source world. As Eghbal explained in a report she produced all the way back in 2016, writing: Nearly all software today relies on free, public code... written and maintained by communities of developers and other talent. Much like roads or bridges, which anyone can walk or drive on, open source code can be used by anyone—from companies to individuals—to build software. This type of code makes up the digital infrastructure of our society today. Just like physical infrastructure, digital infrastructure needs regular upkeep and maintenance. In the United States, over half of government spending on transportation and water infrastructure goes just to maintenance. But financial support for digital infrastructure is much harder to come by.  Tidelift's model, then, ensures that those people who help to maintain the open source software that forms the foundations of our digital lives, are compensated for their work. And it's important to note that this isn't just about charity - the economy's reliance on what is essentially an army of unpaid labor is ultimately unsustainable and makes the broader software ecosystem remarkably fragile. Solving challenges for the organizations that use open source software But it isn't just about open source maintainers. Open source software also poses many challenges for the organizations that actually use it, particularly in terms of reliability and security. At a time when cybersecurity feels like a cat and mouse game between criminals and security experts, and when building resilient distributed systems without downtime is business critical, this becomes incredibly important. Tidelift removes this by incentivising maintainers to work alongside users and ensure that standards are maintained without harming the velocity of innovation that makes open source so attractive. This leads to a level of surprising equilibrium between business and open source maintainers. Going beyond a compromise The tech world has been looking for a compromise when it comes to the challenges of open source. The mission has always been to find a way that pleases business while also ensuring those that maintain it are happy and continue to do it. But if compromise is, as the saying goes, a situation in which all parties are left dissatisfied, Tidelift appears to be going one better by making everyone happy. Tidelift might not be the only solution to some of the problems that open source is facing - we might, for example, see new licensing models begin emerging over time (despite MongoDB's failed attempts to get the SSPL past the OSI) - but it nevertheless looks like a unique and forward-thinking solution to an incredibly important issue. You can learn more about Tidelift by visiting the company's site here.
Read more
  • 0
  • 0
  • 6342

article-image-announcing-docker-enterprise-3-0-public-beta
Savia Lobo
02 May 2019
3 min read
Save for later

Announcing Docker Enterprise 3.0 Public Beta!

Savia Lobo
02 May 2019
3 min read
Update: On July 22, 2019, the Docker team announced that the Docker Enterprise 3.0 will be generally available. He also added that more than 2,000 people have tried the Docker Enterprise 3.0 public beta program On April 24, the team at Docker announced Docker Enterprise 3.0, an end-to-end container platform that enables developers to quickly build and share any type of application (from legacy to cloud-native) and securely run them anywhere, from hybrid cloud to the edge. It is now available in Public Beta Docker Enterprise 3.0 delivers new desktop capabilities, advanced development productivity tools, a simplified and secure Kubernetes stack, and a managed service option to make Docker Enterprise 3.0 the platform for digital transformation. Jay Lyman, the Principal Analyst for 451 Research, “Docker’s new Enterprise 3.0 promises to automate the 'development to production' experience with new tooling that aims to reduce the friction between dev and ops teams.” What can you do with the new Docker Enterprise 3.0? Integrated Docker Desktop Enterprise Docker Desktop Enterprise provides a consistent development-to-production experience with a set of automation tools. This makes it possible to start with the developer desktop, deliver an integrated and secure image registry with access to the Hub ecosystem, and then deploy to an enterprise-ready and Kubernetes-conformant environment. Docker Kubernetes Services (DKS) can simplify the scaling and deployment of applications Compatible with Docker Compose, Kubernetes YAML and Helm charts, DKS provides an automated and repeatable way to install, configure, manage and scale Kubernetes-based applications across hybrid and multi-cloud. DKS includes enhanced security, access controls, and automated lifecycle management bringing a new level of security to Kubernetes that integrates seamlessly with the Docker Enterprise platform. Customers will also have the option to use Docker Swarm Services (DSS) as part of the platform’s orchestration services. Docker Applications for high-velocity innovation Docker Applications are based on the CNAB open standard. It removes the friction between Dev and Ops by enabling teams to collaborate on an application by defining a group of related containers that work together to form an application. It also eliminates the configuration overhead by integrating and automating the creation of the Docker Compose and Kubernetes YAML files, Helm charts, etc. It also includes Application Templates, Application Designer and Version Packs, using which Docker Applications makes it possible for flexible deployment across different environments, delivering on the “code once, deploy anywhere” promise. With the announcement of Docker Enterprise 3.0, Docker also introduced Docker Enterprise-as-a-service - a fully-managed service on-premise or in the cloud. To know more about this news in detail, head over to Docker’s official announcement. DockerHub database breach exposes 190K customer data including tokens for GitHub and Bitbucket repositories Are Debian and Docker slowly losing popularity? Creating a Continuous Integration commit pipeline using Docker [Tutorial]
Read more
  • 0
  • 0
  • 3692

article-image-f8-developer-conference-highlights-redesigned-fb5-app-messenger-update-new-oculus-quest-and-rift-s-instagram-shops-and-more
Bhagyashree R
02 May 2019
5 min read
Save for later

F8 Developer Conference Highlights: Redesigned FB5 app, Messenger update, new Oculus Quest and Rift S, Instagram shops, and more

Bhagyashree R
02 May 2019
5 min read
This week, Facebook hosted F8, its annual developer conference where it made a bunch of announcements about new updates in Facebook family of apps, which includes Messenger, Whatsapp, Instagram, Facebook app, and Oculus. This two-day event started on April 30 and was conducted at San Jose, CA. F8 brings together developers, creators, businesses, partners, and entrepreneurs to network and discuss the future of technology with Facebook and its family of apps. Along with keynotes from Facebook leaders, this event featured sessions, meetups, and demos showcasing the latest in AI, open source, AR/VR, and new tools. Let’s have a look at some of the key announcements Facebook made at this event: Updates in Messenger The Messenger team has overhauled the underlying architecture of Messenger provide users a faster experience. This re-engineered version is planned to come out later this year. Following are some of the features the updated Messenger will come with: A feature is introduced that allows users to share Facebook videos on Messenger and invite family and friends to watch them together in real-time. The Messenger Desktop app is now available for macOS and Windows users, with which users can have group video calls or collaborate on projects. Instagram’s “Close Friends” feature is now coming to the new Messenger for sharing stories and messaging your closest friends and family. The team is adding lead generation templates to Ads Manager to allow businesses to easily reach out to potential customers. Show your business catalog on Whatsapp Source: Facebook Many businesses and customers find WhatsApp a secure and comfortable place to connect and converse. Whatsapp is now making business much easier by introducing a feature for showing business catalog. With this feature, businesses can showcase their products within Whatsapp and also chat with them about the orders. Redesigned Facebook app Source: Facebook The Facebook app is now revamped with a “fresh new design” and is called the “FB5” version. The design is simpler, faster, and puts the main focus on Groups. The app introduces new tools to make it easier for users to find and interact with groups of people who have common interests and passions. This FB5 version comes with few tools that make it easier for users to discover new groups that they might be interested in. It has a redesigned Groups tab that shows a personalized feed of activity across all your groups. A new discovery tool will show improved recommendations that allow you to quickly find groups you might be interested in. The team is also working to show relevant group recommendation in other parts of the app as well like Marketplace. The app will also come with new features for a specific type of group. For instance, Health Support groups will allow its members to post questions and share information anonymously, Job groups will have a new template for employers to post new job openings and more. Matching with your “Secret Crush” Facebook Dating, the opt-in service to find your potential matches is now coming to 14 more countries. These include the Philippines, Vietnam, Singapore, Malaysia, Laos, Brazil, Peru, Chile, Bolivia, Ecuador, Paraguay, Uruguay, Guyana, and Suriname. Facebook Dating will now also support a new feature called Secret Crush with which you can express your interest in someone you like. All you have to do is just select up to nine of your Facebook friends who you want to express interest in. If your crush has opted into Facebook Dating, they will be notified that someone has a crush on them. If your crush also adds you to their Secret Crush list it is a match! Portal and Portal Plus coming to Europe After launching Portal and Portal Plus smart displays in the US and Canada last year, Facebook plans to make them available in Europe starting from this fall. Portal will come with support for Whatsapp to allow users make calls to friends who are also using WhatsApp or Messenger on their Portal or on their phone. These calls will be end-to-end encrypted. Along with supporting video calls, Portal will also come with some AI assistant features. Starting from this summer, to get updates on birthdays, events and more you just have to say “Hey Portal, Good Morning”. You will also be able to send private video messages from Portal to your loved ones. The team has collaborated with Amazon to bring Alexa skills to Portal including Flash Briefings, smart home control and the Amazon Prime Video app later this year. Oculus Quest and Rift S available for pre-orders Facebook’s latest VR headsets, Oculus Quest and Oculus Rift S will ship on May 21, both priced at $399 USD. Oculus Quest allows you to play almost anywhere with just a VR headset and controllers. Rift S is the PC-powered gaming headset, which is a successor of the classic Oculus VR headsets. Shop and raise funds on Instagram Instagram will soon allow users to shop looks shared by their creators without leaving Instagram. You will no longer have to ask for product details in comments or DM, just tap on the picture to see the details and buy it on the spot. Users can also raise money for any social cause they care about directly on Instagram. You can do this through a donation sticker in Stories and send 100% of the raised money directly to the non-profit you are supporting. This feature is currently available only in the US and will soon be supported in other countries as well. Read the official announcement by Facebook for more details. Facebook open-sources F14 algorithm for faster and memory-efficient hash tables “Is it actually possible to have a free and fair election ever again?,” Pulitzer finalist, Carole Cadwalladr on Facebook’s role in Brexit New York AG opens investigation against Facebook as Canada decides to take Facebook to Federal Court for repeated user privacy violations
Read more
  • 0
  • 0
  • 3901

article-image-silicon-valley-investors-envy-chinas-996-work-culture-but-tech-workers-stand-in-solidarity-with-their-chinese-counterparts
Sugandha Lahoti
01 May 2019
9 min read
Save for later

Silicon Valley investors envy China’s 996 work culture; but tech workers stand in solidarity with their Chinese counterparts

Sugandha Lahoti
01 May 2019
9 min read
The news of China's inhumane working hour policies is in the spotlight in the high growth tech world. Last month, a Github user with the name “996icu” created a webpage that he shared on GitHub, to protest against the “996” work culture in Chinese tech companies. The “996” work culture is an unofficial work schedule that requires employees to work from 9 am to 9 pm, 6 days a week, totaling up to 60 hours of work per week. In fact, such is the culture, that Chinese tech recruiters are instructed by their bosses to not hire people over 30 years of age. Close to three-quarters of tech workers in China are below thirty in age and the employers further promote this concept, per a Bloomberg post published by Shelly Banjo, Roving Asia Tech Reporter. 996 amassed more fire when Jack Ma, co-founder and executive chairman of the Alibaba Group, the Chinese rival to Amazon, defended the 12-hour, 6-day week working schedule and chastised people wanting a balanced and typical eight-hour work shift. In his own words, “To be able to work 996 is a huge blessing. If you want to join Alibaba, you need to be prepared to work twelve hours a day, otherwise why even bother joining.” Although the 996 culture violates Labor Law of the People’s Republic of China, and was the point of scrutiny and condemn by a large number of publishing firms, tech activists, and developers, a few are still favoring this inhumane culture. These are the so-called Silicon Valley investors and founders who are so jealous of China's tech space that they are promoting other employers to follow China’s footsteps. Silicon Valley Investors envy the Chinese 996 culture It seems that regardless of the chastising and condemnation, the 996 culture has got the Silicon Valley spooked. Their argument, China’s 996 schedule is the kind of work ethic that will eventually help it become the superpower defeating the US. Something that all US founders and investors should aspire for. “Founders: We're up against JackMa (& China) *enforcing* a 72-hour work-week 996 = 6 days a week, 9 am to 9 pm. The same exact work ethic that built America! You can get on your twitter pedestal & attack Ma or you can make a plan to win.” This is a tweet posted by Jason Calacanis, an Angel investor for Uber, Thumbtack, Wealthfront, and more companies. He further adds, “Not going to tell you how many hours you should work, but I will tell you that you need a plan to fight heads up against Chinese companies -- which will be going head-to-head with every startup every created in the next decade. Ignoring, deriding or dismissing China isn't a plan.” In January, Mike Moritz, a venture capitalist of Sequoia Capital wrote an editorial in the Financial Times titled “Silicon Valley would be wise to follow China’s lead.” He wrote that Silicon Valley has become “unhinged” with discussions about the “inequity of life.” He contrasted Silicon Valley tech culture with the work ethic in Chinese tech companies, where employees work for 14 hours six or seven days a week. He adds, “in many respects, doing business in China is easier than doing business in California.” Other quotes from Moritz’s piece speaks volumes about the inhumane 996 culture and how Chinese employees recuperate with the reality of such extensive work hours. "Top managers show up for work at about 8 am and frequently don't leave until 10 pm. Most of them will do this six days a week — and there are plenty of examples of people who do this for seven. Engineers have slightly different habits: they will appear about 10 am and leave at midnight. Beyond the week-long breaks for Chinese new year and the October national holiday, most will just steal an additional handful of vacation days. Some technology companies also provide a rental subsidy to employees who choose to live close to corporate HQ. In China, by contrast, it is quite usual for the management of 10 and 15-year-old companies to have working dinners followed by two or three meetings. Many of these high-flyers only see their children — who are often raised by a grandmother or nanny — for a few minutes a day.” Moritz’s argument and his support for China’s 996 culture sparked a lively discussion on Quora. It has also invited a tsunami of unsurprisingly negative responses across the Silicon Valley tech community. Andy Manoske, former Associate at GGV Capital penned a blog post condemning Moritz’s condescending statements. Andy says, “Mike’s comments on paternity leave and seemingly slavish dedication run contrary to empirical evidence of the success of maintaining work life balance. Mike himself (an investor who has never actually worked in Silicon Valley as an operator and joined Sequoia after being a journalist) seems out of touch with the brutal realities of Silicon Valley’s already-imbalanced work-life balance, and that his post is clearly just an attempt to “win points” with Chinese tech firms that Sequoia is courting.” He further adds, “I feel this was a haphazard piece that read like a desperate attempt to somewhat patronizingly woo Chinese startups (most of whom will not read the Financial Times) and seemingly establish Western credibility on China.” One of the Quorans, Bowen Li wrote, “I think somebody read their history book in reverse order. Societies progress by giving people more rights and freedoms over time, not less. Creative, information work - the most valuable type of work that happens in Silicon Valley - does not benefit from increased hours. You cannot create twice as much innovation by putting in twice the hours. The human brain simply fatigues past a certain point and you get diminished or negative returns. People aren’t machines; thinking of this type of highly skilled work as “hours in, product out” is simply the wrong way to look at it.” According to job-hunting site Maimai, reports Reuters, the tech sector was the only industry out of thirteen surveyed to see more people leave than join between October 2018 and February 2019. This means 996 can result in an additional cost for tech firms, venture capitalists and analysts instead of increasing productivity. “One of the highest costs in an organization is high employee turnover. A culture that is less focused on hours put in, may also become more effective if the focus is turned to output versus input,” said Rui Ma, a San Francisco-based investor who has funded startups in China and North America. Tech solidarity and grassroots mobilization is necessary to eradicate 996 It is the tech workers solidarity which needs to fleshed out in order to eliminate the inhumane 996 culture. The tech worker groups should unionize and be backed by political campaigns, advocacy groups, and nonprofits to start a political conversation and effect change at an international level. This has already begun with Chinese state media newspaper People’s Daily criticizing the 996 culture, In an editorial they wrote, “Employees who object to 996 cannot be labeled as ‘slackers’ or ‘not fighters’. Their real needs should be considered.” Recently, Microsoft and GitHub employees also drafted a petition in defense of the GitHub repository which could be under threat of Chinese censorship. The project is an initiative towards making the Chinese tech companies obey the labor laws and the international labor convention. The 996.ICU GitHub project description reads, “By following the ‘996’ work schedule, you are risking yourself getting into the ICU (Intensive Care Unit).” This petition was signed by 50 tech employees altogether, including several from Google, urging Microsoft and GitHub not to remove the 996.ICU project from the GitHub site. “We, the workers of Microsoft and GitHub, support the 996.ICU movement and stand in solidarity with tech workers in China. We know this is a problem that crosses national borders. These same issues permeate across full time and contingent jobs at Microsoft and the industry as a whole," the letter said. https://twitter.com/MsWorkers4/status/1120809928577306624 https://twitter.com/techworkersco/status/1120791636282187776 https://twitter.com/chinalaborwatch/status/1114197666530058241 Trade Unions should also come forward to support employees enduring 996 hardships. Discussing #996ICU Shen Jianfeng of the China University of Labor Relations, noted in the Global Times that  "trade unions in China should indeed play an active role in safeguarding the rights and interests of workers" in the tech sector. Other software developers also rose in mutual support of Chinese developers and more should do so. https://twitter.com/rwfholme/status/1121136973916151808 https://twitter.com/SlyFireFox/status/1120825434558619648 The US tech companies need to awaken to the Chinese competition but worrying about China’s 996 culture is not one of them. On the surface, working 996 is about companies working as many hours of the week as possible in order to beat the competition and to capture a market before the competition. However, a lot of other factors such as quality of the working environment, worker’s exposure to stress and the ability to frequently rest well actually determines a company’s progress. Working endlessly can make an individual less effective than if they work for fewer hours in a calmer manner. Not only work, but the 996 schedule also deprives people of their free time and makes families and relationships suffer, for no extra pay and no extra output. To conclude, Silicon Valley can, and should, learn lessons from Chinese tech cultures. But none of these should have to do with oppressive and authoritarian management philosophies like the unhealthy dedication to work or human rights eroding business decisions such as mass surveillance systems that are prevalent in China. ‘Developers’ lives matter’: Chinese developers protest over the “996 work schedule” on GitHub Jack Ma defends the extreme “996 work culture” in Chinese tech firms What the US-China tech and AI arms race means for the world – Frederick Kempe at Davos 2019
Read more
  • 0
  • 0
  • 7132

article-image-notokgoogle-employee-led-town-hall-reveals-hundreds-of-stories-of-retaliation-at-google
Sugandha Lahoti
27 Apr 2019
6 min read
Save for later

#NotOkGoogle: Employee-led town hall reveals hundreds of stories of retaliation at Google

Sugandha Lahoti
27 Apr 2019
6 min read
Note: This article now includes stories of various other Google employees about the retaliation they faced, which they have shared with Google Walkout for real change. Last week, two Google Walkout organizers accused the company of retaliation against them over last year’s Google Walkout protest. The two Google employees, Claire Stapleton, YouTube Marketing Manager and Meredith Whittaker, head of Google’s Open Research were told their roles would change dramatically including calls to abandon AI ethics work, demotion, and more. In regards to this, both women hosted a Retaliation Town Hall to share their stories and strategize this Friday. The event was also live-streamed to other Google offices. 350+ Google staffers attended the Town Hall, where Stapleton said, that after her public sharing of retaliation earlier this week, Google executives tried to undermine her “in emails this week to thousands of my friends and colleagues”. In those emails, Google executives challenged Stapleton’s claim saying that she was “never demoted”. “What’s true is that there was a demotion, and after my lawyers got involved, it was reversed,” Stapleton wrote, in a statement viewed by Wired. Whittaker also provided further updates on the retaliation she faced. She said that her manager told her in late December she would likely need to leave Google’s Cloud division. The same manager told her in March that the “Cloud division was seeking more revenue and that AI Now and her AI ethics work was no longer a fit. This was a strange request because the Cloud unit has a team working on ethical concerns related to AI.” During the March meeting, Whittaker claims her manager also told her that there are “two kinds of people at Google...those who quit, and those who stay and hate every minute of it and try to destroy it. I was taken aback, since my work has always aimed to ensure that Google lived up to its purported values and treated its workforce, and the rest of the world, with respect.” Recently, Google dissolved it’s AI ethics council after nearly 2,600 employees, including Whittaker,  signed a petition against the appointment of Kay Cole James, president of the Heritage Foundation. Employees were upset by James’ anti-trans and anti-immigrant political statements. Whittaker also signed the petition protesting Google’s infamous Project Dragonfly, the secretive search engine that Google is allegedly developing which will comply with the Chinese rules of censorship. Meredith Whittaker was also a leader in the anti-Maven movement. Google’s Project Maven, was focused on analyzing drone footage and could have been eventually used to improve drone strikes on the battlefield. More than 3,000 Google employees signed a petition against this project that led to Google deciding not to renew its contract with the U.S. Department of Defense in 2019. In her statement on Friday, Whittaker says that on April 1, a few days before the petition was drafted, she got approval from Jeff Dean to transfer from Cloud to Google’s Research and Machine Intelligence group. Two weeks after the petition was sent, Whittaker claims her transfer was killed. Other Google employees also spoke about employee discontent at Google ranging from the ethics of performing work for the US Department of Defense to the handling of sexual harassment claims. Following the statements of Whittaker and Stapleton in the town hall session, several current and former Googlers took to Twitter to register complaints and share their experiences of facing retaliation from the company. They expressed their disagreement with Google’s policies with the hashtag #NotOkGoogle which was trending on Friday. https://twitter.com/mer__edith/status/1121789412776525824 “This does not seem to be an isolated incident”, Vanessa Harris, Google Product Manager https://twitter.com/technologypoet/status/1121953562232098817 “I am grateful that I quit Google”, Liz Fong-Jones, ex-Googler, current SRE Dev Advocate @honeycombio https://twitter.com/lizthegrey/status/1121850362158178304 She also talked about Google's retaliation against her which forced her to quit. https://twitter.com/lizthegrey/status/1120373545274445831 “This is just the tip of the iceberg", Dr. Alex Hanna, a computational social scientist at Google Cloud https://twitter.com/alexhanna/status/1121848713037500416 Mila Hardt from Google Health division https://twitter.com/Mi_Mo/status/1121849275153915904 “I emailed @EileenTNaughton later again in November, just after the town hall meeting and her promises for change. I pledged her again to help stop the planned disposal of me. This time she never responded! My last day was after Thanks Giving! Thanks, Google for the gift!”, Vida Vakilotojar, Xoogler https://twitter.com/VidaVakil/status/1121995359012573184 The town hall group also published an internal document with a new set of “demands”. The document which was seen by The Guardian includes a “transparent, open investigation of HR and its abysmal handling of employee complaints relating to working conditions, discrimination, harassment, and retaliation”. Other demands include a public response from Google co-founder Larry Page, and that Google meets the demands that were issued in the Google walkout. “Google has had six months to meet [those] demands; in that time, they’ve partially met only one of them,” the document states. “Google seems to have lost its mooring and trust between workers and the company is deeply broken. The company has no clear direction and is just progressing from crisis to crisis lately.” Google did not respond to specific questions about the town hall’s meeting. A spokeswoman said in a statement: “We prohibit retaliation in the workplace and publicly share our very clear policy. To make sure that no complaint raised goes unheard at Google, we give employees multiple channels to report concerns, including anonymously, and investigate all allegations of retaliation.” Other activist and worker groups also rose in solidarity. https://twitter.com/TIMESUPNOW/status/1121834802284355584 “The impact @mer__edith has in AI ethics is second to none. What happens to her at Google will be a gauge for the wellbeing of the entire field. Watch closely.”, Moritz Hardt, Assistant Professor of Electrical Engineering and Computer Science, Berkeley University https://twitter.com/mrtz/status/1121110692843507712 Update: Yesterday, Google Walkout for real change published a blog post on medium sharing stories of retaliation from various other Google employees. "When I reported something unethical happening at Google, Employee Relations fudged data to protect Google." "Retaliated against for defending a mother who reports to me. HR dismissed it as “poor behavior" "I reported my tech lead to my manager for sexual harassment, but my manager thought I was 'overreacting' " "My first two years at Google I was not promoted due to bias. While my peer was promoted with the same ratings and same tenure, I was not. When I asked my manager about this I was told that I was being an “emotional woman.” How is this happening at Google? Clearly this pattern should not be allowed to continue. Google announces new policy changes for employees to report misconduct amid complaints of retaliation and harassment. #GoogleWalkout organizers face backlash at work, tech workers show solidarity Google employees ‘Walkout for Real Change’ today. These are their demands.
Read more
  • 0
  • 0
  • 4082

article-image-best-practices-in-go-gui-development-tutorial
Amrata Joshi
27 Apr 2019
12 min read
Save for later

Best Practices in Go GUI Development [Tutorial]

Amrata Joshi
27 Apr 2019
12 min read
Separation of concerns is closely related to the Single Responsibility Principle introduced by Robert C. Martin in his principles of Object Oriented Design, which state that: "A class should have one, and only one, reason to change." –Robert C. Martin In this respect, concerns have a wider scope than responsibilities, typically influencing your application's design and architecture rather than individual classes or interfaces. Separation of concerns is essential in a graphical application to correctly detach your easily-tested logic from the presentation code, which manages user interaction. This article is an excerpt taken from the book Hands-On GUI Application Development in Go. This book covers the benefits and complexities of building native graphical applications, the procedure for building platform and developing graphical Windows applications using Walk.  In this article, we will learn certain aspects of best practices that make it easier to maintain and grow GUI-based applications. This article covers separation of concerns, test-driving UI development and much more. By separating the concerns of an application, it is easier to test subcomponents and check the validity of our software without even needing to run the application. In doing so, we create more robust applications that can adapt to changes in requirements or technology over time. For example, the graphical toolkit that you choose for your application should not be incorporated into, or impact the design of, your business logic. Suggested application structure As you plan the development of your application, consider how the core concerns could be separated to maintain flexibility. The following suggested structure should provide some inspiration: project/ The root of the project structure. This package should define the interfaces and utility functions used by the rest of the project. These files should not depend on any sub-packages. project/logic/ This package will contain most of your application logic. Careful consideration should be given to which functions and types are exposed, as they will form the API that the rest of your application will depend upon. There may be multiple packages that contain application logic as you separate the application's concerns. An alternative, domain-specific term may be preferred to logic. project/storage/ Most applications will rely upon a data source of some kind. This package will define one or many possible data sources. They will conform to an interface in the top-level project so that data access can be passed between packages of the project. project/gui/ This package is the only place where your graphical toolkit should be imported. It is responsible for loading your application GUI and responding to user events. It will probably access data provided by a storage package set from the application runner. project/cmd/appname/ The Go convention for application binaries is that they reside within a cmd/appname sub-package. The actual package for this directory will be main, and it will contain, minimal code that is required to load and run the main application defined within the other packages. It will probably initialize a storage system, load the application logic, and instruct the graphical interface to load. When writing tests in each of these packages, they will focus on the functionality of the current package. The logic package should have very high unit-test coverage, whereas the storage package may rely more on integration testing. The gui package, which is often considered the hardest to test, could directly import the logic package in its tests, but should probably not include the main storage package to validate its functionality. Following a sensible structure will aid significantly in making your application testable, as many developers are probably already aware. It is often much harder, however, to test the graphical portions of an application. Designing your application to be unit-testable from the beginning will often result in a code base that is better organized and will naturally lead to code that is easier to understand and change. Let's take a look at what Test-driven Development (TDD) can teach us about building graphical interfaces. Test-driving UI development The effort required to automatically test user interfaces or frontend software is often debated as being far too expensive for the value it returns in avoiding future bugs. However, this is largely rooted in the toolkits being utilized or even the presentation technologies chosen. Without full support for testing in the development tools or graphical APIs, it can indeed be difficult to create simple unit tests without a huge effort. As seen frequently in web-based environments (and some native test frameworks), the only remaining possibility is to run the application and execute test scripts that will perform the validation. They will typically control the user input, simulating mouse actions and keyboard taps, and monitor the resulting behavior of the application under test. If, however, your application and GUI toolkit are architected with testing in mind (for example, using separation of concerns), automated tests should be possible with far less overhead. Designed to be testable When setting out the components within a project's UI code (as illustrated in the gui sub-package), care should be taken to define types and classes that have a single responsibility and a clear API. Doing so will make it easier to load and test individual components with the standard Go testing tools. If smaller components can be tested, we can avoid launching the whole application and the required test runners, therefore making the testing process much faster. When a test suite runs quickly, it can be run more frequently and extended more easily, leading to higher test coverage and greater confidence in the software quality. For a practical example, let's look at the GoMail compose dialog and its Send button. Clearly, the dialog box should perform all sorts of validation before sending, and if they pass then send the email. Validation can easily be tested with normal unit tests, but verifying that the send button correctly sends a new email will require the user interface to be tested. In the following example, we will load the compose window, enter some data, and simulate the Send button being pressed. By using a test email server, as used through each of the GoMail examples, we can check that the email has been sent by the user interface without needing to communicate with a real email server. Example application test As the tests are in the same package, we can test internal function definitions rather than relying on exported APIs—this is common with UI code as long as the application is not large enough to warrant separate packages or libraries. We start by adding the test imports; testing is required for go test code and github.com/stretchr/testify/assert provides helpful assertion functionality. We also import the client email library created for our GoMail examples and finally the Fyne test package, fyne.io/fyne/test: package main import ( "testing" "fyne.io/fyne/test" "github.com/PacktPublishing/Hands-On-GUI-Application-Development-in-Go/client" "github.com/stretchr/testify/assert" ) Now we can add a test method using the recommended naming pattern of Test<type>_<function>(); normally, the function would be a function name, but here we refer to the button title or its action. In the first part of the function, we set up the compose window for testing by calling newCompose() and passing it a test application (returned from test.NewApp()). We then prepare the state for our test—we record the size of the server outbox and set up an OnClosed handler that will report when the window is closed. Finally, we simulate typing an email address into the compose.to field using test.Type(): func TestCompose_Send(t *testing.T) { server := client.NewTestServer() compose := newCompose(test.NewApp(), server) ui := compose.loadUI() pending := len(server.Outbox) closed := false ui.SetOnClosed(func() { closed = true }) address := "test@example.com" test.Type(compose.to, address) ... } Once the setup code is complete, we can implement the main test. This starts by using test.Tap() to tap the compose.send button, which should cause an email to be sent. We first verify that the window was closed after the email send completes (the OnClosed handler we added records this). Then we check that there is one more email in the server.Outbox than before. If these tests pass, we will move to the final check. The email that was sent is extracted from the outbox so we can examine its content. With one final assertion, we verify that the email address matched what we typed into the To input box: func TestCompose_Send(t *testing.T) { ... test.Tap(compose.send) assert.True(t, closed) assert.Equal(t, pending + 1, len(server.Outbox)) email := server.Outbox[len(server.Outbox)-1] assert.Equal(t, address, email.ToEmailString()) } Running the preceding test will load the user interface in memory, execute the setup code, and run the tests, and then exit with the results. We run the following test with -v to see each test that is run rather than just a summary. You will notice that testing in this way takes very little time (go test reports 0.00 seconds for the test and 0.004 seconds in total); therefore, many more tests could be run on a regular basis to verify the application's behavior: Running the user interface test took very little time When running the tests, you may notice that this test does not cause any window to be displayed on your computer screen. This is a design feature of many test frameworks for GUI toolkits – it is much faster to run the application without displaying it for test purposes. This is often called headless mode and is very useful when running automated tests as part of a continuous integration process. Continuous integration for GUIs Continuous integration (the regular merging of a team's work-in-progress code to be automatically tested) has become commonplace in software development teams. Adding this process to your team workflow is shown to highlight issues earlier in the development process, which leads to fixing issues faster and, ultimately, better-quality software. A critical part of this is the automation of tests that exercise the whole of the source code, which includes the graphical user interface. Approaches to GUI test automation It is important to organize your code into logical components for development and testing. Using the framework test features (or external support libraries) smaller components can more easily be verified through simple tests. The Go language's built-in support for testing has meant that test coverage is improving; in fact, the popular Go library list, on Awesome, asks that libraries have a test coverage of at least 80%! If your chosen framework does not provide the necessary support, it is still possible to automate functional testing. The technique involves running the application from a test script that then performs simulated user actions on the host computer. This is not ideal as it requires the application to be visible on the screen and for the test script to take control of the keyboard and mouse – but it is better than having no GUI testing in place. To work around this inconvenience, it is possible to load a virtual frame buffer (an off-screen display area) in which to run the application. This technique basically creates an invisible screen to which the application can draw. Avoiding external dependencies One thing to be aware of when testing an application, or portions of it, is that there may be external systems involved. A file browser may rely on network connections for some of its work, or an instant messenger app is going to need a server to handle sending and receiving messages. If your code has been organized carefully to separate its concerns, you will already have used interfaces to define the interactions between different components. If this approach is taken, we can use dependency injection to provide alternative implementations for areas of an application that should not be included in automated testing. When code is properly decoupled from the components that it relies on, it's possible to load different versions of an application for testing. In this manner, we can avoid relying on any external systems or causing permanent changes to a data store. Let's look at a trivial example, a Storage interface is defined that will be used to read and write files from a disk: type Storage interface { Read(name string) string Write(name, content string) } There is an application runner that invokes permanent storage and uses it to write and then read a file: func runApp(storage Storage) { log.Println("Writing README.txt") storage.Write("README.txt", "overwrite") log.Println("Reading README.txt") log.Println(storage.Read("README.txt")) } func main() { runApp(NewPermanentStorage()) } Clearly, this application will cause whatever was in an existing README.txt file to be overwritten with the contents of overwrite. If we assume, for example, that this is the desired behavior, we probably don't want this external system (the disk) to be affected by our tests. Because we have designed the storage to conform to an interface, our test code can include a different storage system that we can use in tests, as follows: type testStorage struct { items map[string]string } func (t *testStorage) Read(name string) string { return t.items[name] } func (t *testStorage) Write(name, content string) { t.items[name] = content } func newTestStorage() Storage { store := &testStorage{} store.items = make(map[string]string) return store } Following this addition, we can test our application's runApp function without the risk of overwriting real files: import ( "testing" "github.com/stretchr/testify/assert" ) func TestMain_RunApp(t *testing.T) { testStore := newTestStorage() runApp(testStore) newFile := testStore.Read("README.txt") assert.Equal(t, "overwrite", newFile) } When running this test, you will see that we get the expected result, and should also notice that no real files have changed. See that our TestMain_RunApp completed successfully without writing to our disk In this article, we explored some of the tips and techniques for managing a GUI-based application written with Go. We have learned about the separation of concerns, test-driving UI development and much more. To know more about managing specific platforms with Go, check out the book Hands-On GUI Application Development in Go. GitHub releases Vulcanizer, a new Golang Library for operating Elasticsearch State of Go February 2019 – Golang developments report for this month released The Golang team has started working on Go 2 proposals
Read more
  • 0
  • 0
  • 8720
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-new-york-ag-opens-investigation-against-facebook-as-canada-decides-to-take-facebook-to-federal-court-for-repeated-user-privacy-violations
Savia Lobo
26 Apr 2019
6 min read
Save for later

New York AG opens investigation against Facebook as Canada decides to take Facebook to Federal Court for repeated user privacy violations

Savia Lobo
26 Apr 2019
6 min read
Despite Facebook’s long line of scandals and multiple parliamentary hearings, the company and its leadership have remained unscathed, with no consequences or impact on their performance. Once again, Facebook is under fresh investigations; this time from New York’s Attorney General, Letitia James. The Canadian and British Columbia privacy commissioners have also decided to take Facebook to Federal Court to seek an order to force the company to correct its deficient privacy practices. It remains to be seen if Facebook’s lucky streak would continue in light of these charges. NY Attorney General’s investigation over FB’s email harvesting scandal Yesterday, New York’s Attorney General, Letitia James opened an investigation into Facebook Inc.’s unauthorized collection of 1.5 million users’ email contacts without users’ permission. This incident, which was first reported on Business Insider, happened last month where Facebook’s email password verification process for new users asked users to hand over the password to their personal email account. According to the Business Insider report, “a pseudononymous security researcher e-sushi noticed that Facebook was asking some users to enter their email passwords when they signed up for new accounts to verify their identities, a move widely condemned by security experts.” https://twitter.com/originalesushi/status/1112496649891430401 Read Also: Facebook confessed another data breach; says it “unintentionally uploaded” 1.5 million email contacts without consent On March 21st, Facebook opened up about a major blunder of exposing millions of user passwords in a plain text, soon after Security journalist, Brian Krebs first reported about this issue. “We estimate that we will notify hundreds of millions of Facebook Lite users, tens of millions of other Facebook users, and tens of thousands of Instagram users”, the company said in their press release. Recently, on April 18, Facebook updated the same post stating that not tens of thousands, but “millions” of Instagram passwords were exposed. “Reports indicate that Facebook proceeded to access those user’s contacts and upload all of those contacts to Facebook to be used for targeted advertising”, the Attorney General mentioned in the statement. https://twitter.com/NewYorkStateAG/status/1121512404272189440 She further mentions that “It is time Facebook is held accountable for how it handles consumers' personal information.” “Facebook has repeatedly demonstrated a lack of respect for consumers’ information while at the same time profiting from mining that data. Facebook’s announcement that it harvested 1.5 million users’ email address books, potentially gaining access to contact information for hundreds of millions of individual consumers without their knowledge, is the latest demonstration that Facebook does not take seriously its role in protecting our personal information”, James adds. “Facebook said last week that it did not realize this collection was happening until earlier this month when it stopped offering email password verification as an option for people signing up to Facebook for the first time”, CNN Business reports. One of the users on HackerNews wrote, “I'm glad the attorney general is getting involved. We need to start charging Facebook execs for these flagrant privacy violations. They're being fined 3 billion dollars for legal expenses relating to an FTC inquiry… and their stock price went up by 8%. The market just does not care; it's time regulators and law enforcement started to.” To know more about this news in detail, read Attorney General James’ official press release. Canadian and British Columbia privacy commissioners to take Facebook to Federal Court Canada and British Columbia privacy commissioners Daniel Therrien and Michael McEvoy, uncovered major shortcomings in Facebook’s procedures in their investigation, published yesterday. This investigation was initiated after media reported that “Facebook had allowed an organization to use an app to access users’ personal information and that some of the data was then shared with other organizations, including Cambridge Analytica, which was involved in U.S. political campaigns”, the report mentions. The app, at one point, called “This is Your Digital Life,” encouraged users to complete a personality quiz. It collected information about users who installed the app as well as their Facebook “friends.” Some 300,000 Facebook users worldwide added the app, leading to the potential disclosure of the personal information of approximately 87 million others, including more than 600,000 Canadians. The investigation also revealed that Facebook violated federal and B.C. privacy laws in a number of respects. According to the investigation, “Facebook committed serious contraventions of Canadian privacy laws and failed to take responsibility for protecting the personal information of Canadians.” According to the press release, Facebook has disputed the findings and refused to implement the watchdogs’ recommendations. They have also refused to voluntarily submit to audits of its privacy policies and practices over the next five years. Following this, the Office of the Privacy Commissioner of Canada (OPC) said it, therefore, plans to take Facebook to Federal Court to seek an order to force it the company to correct its deficient privacy practices. Daniel Therrien, the privacy commissioner of Canada, said, “Facebook’s refusal to act responsibly is deeply troubling given the vast amount of sensitive personal information users have entrusted to this company. Their privacy framework was empty, and their vague terms were so elastic that they were not meaningful for privacy protection.” He further added, “The stark contradiction between Facebook’s public promises to mend its ways on privacy and its refusal to address the serious problems we’ve identified – or even acknowledge that it broke the law – is extremely concerning. It is untenable that organizations are allowed to reject my office’s legal findings as mere opinions.” British Columbia Information and Privacy Commissioner Michael McEvoy said, “Facebook has spent more than a decade expressing contrition for its actions and avowing its commitment to people’s privacy. But when it comes to taking concrete actions needed to fix transgressions they demonstrate disregard.” The press release also mentions that “giving the federal Commissioner order-making powers would also ensure that his findings and remedial measures are binding on organizations that refuse to comply with the law”. To know more about the federal and B.C. privacy laws that FB violated, head over to the investigation report. Facebook AI introduces Aroma, a new code recommendation tool for developers Ahead of Indian elections, Facebook removes hundreds of assets spreading fake news and hate speech, but are they too late? Ahead of EU 2019 elections, Facebook expands its Ad Library to provide advertising transparency in all active ads
Read more
  • 0
  • 0
  • 2751

article-image-datacamp-reckons-in-metoo-movement-ceo-steps-down-from-his-role-indefinitely
Fatema Patrawala
25 Apr 2019
7 min read
Save for later

DataCamp reckons with its #MeToo movement; CEO steps down from his role indefinitely

Fatema Patrawala
25 Apr 2019
7 min read
The data science community is reeling after data science learning startup DataCamp penned a blog post acknowledging that an unnamed company executive made "uninvited physical contact" with one of its employees. DataCamp, which operates an e-platform where aspiring data scientists can take courses in coding and data analysis is a startup valued at $184 million. It has additionally raised over $30 million in funding. The company disclosed in a blog post published on 4th April that this incident occurred at an "informal employee gathering" at a bar in October 2017. The unnamed DataCamp executive had "danced inappropriately and made uninvited physical contact" with the employee on the dance floor, the post read. The company didn't name the executive involved in the incident in its post. But called the executive's behavior on the dance floor "entirely inappropriate" and "inconsistent" with employee expectations and policies. When Buisness Insider reached out to one of the course instructors OS Keyes familiar with this matter, Keyes said that the executive in question is DataCamp's co-founder and CEO Jonathan Cornelissen. Yesterday Motherboard also reported that the company did not adequately address sexual misconduct by a senior executive there and instructors at DataCamp have begun boycotting the service and asking the company to delete their courses following allegations. What actually happened and how did DataCamp respond? On April 4, DataCamp shared a statement on its blog titled “a note to our community.” In it, the startup addressed the accusations against one of the company’s executives: “In October 2017, at an informal employee gathering at a bar after a week-long company offsite, one of DataCamp’s executives danced inappropriately and made uninvited physical contact with another employee while on the dance floor.” DataCamp got the complaint reviewed by a “third party not involved in DataCamp’s day-to-day business,” and said it took several “corrective actions,” including “extensive sensitivity training, personal coaching, and a strong warning that the company will not tolerate any such behavior in the future.” DataCamp only posted its blog a day after more than 100 DataCamp instructors signed a letter and sent it to DataCamp executives. “We are unable to cooperate with continued silence and lack of transparency on this issue,” the letter said. “The situation has not been acknowledged adequately to the data science community, leading to harmful rumors and uncertainty.” But as instructors read the statement from DataCamp following the letter, many found the actions taken to be insufficient. https://twitter.com/hugobowne/status/1120733436346605568 https://twitter.com/NickSolomon10/status/1120837738004140038 Motherboard reported this case in detail taking notes from Julia Silge, a data scientist who co-authored the letter to DataCamp. Julia says that going public with our demands for accountability was the last resort. Julia spoke about the incident in detail and says she remembered seeing the victim of the assault start working at DataCamp and then leave abruptly. This raised “red flags” but she did not reach out to her. Then Silge heard about the incident from a mutual friend and she began to raise the issue with internal people at DataCamp. “There were various responses from the rank and file. It seemed like after a few months of that there was not a lot of change, so I escalated a little bit,” she said. DataCamp finally responded to Silge by saying “I think you have misconceptions about what happened,” and they also mentioned that “there was alcohol involved” to explain the behavior of the executive. DataCamp further explained that “We also heard over and over again, ‘This has been thoroughly handled.’” But according to Silge and other instructors who have spoken out, say that DataCamp hasn’t properly handled the situation and has tried to sweep it under the rug. Silge also created a private Slack group to communicate and coordinate their efforts to confront this issue. She along with the group got into a group video conference with DataCamp, which was put into “listen-only” mode for all the other participants except DataCamp, meaning they could not speak in the meeting, and were effectively silenced. “It felt like 30 minutes of the DataCamp leadership saying what they wanted to say to us,” Silge said. “The content of it was largely them saying how much they valued diversity and inclusion, which is hard to find credible given the particular ways DataCamp has acted over the past.” Following that meeting, instructors began to boycott DataCamp more blatantly, with one instructor refusing to make necessary upgrades to her course until DataCamp addressed the situation. Silge and two other instructors eventually drafted and sent the letter, at first to the small group involved in accountability efforts, then to almost every DataCamp instructor. All told, the letter received more than 100 signatures (of about 200 total instructors). A DataCamp spokesperson said in response to this, “When we became aware of this matter, we conducted a thorough investigation and took actions we believe were necessary and appropriate. However, recent inquiries have made us aware of mischaracterizations of what occurred and we felt it necessary to make a public statement. As a matter of policy, we do not disclose details on matters like this, to protect the privacy of the individuals involved.” “We do not retaliate against employees, contractors or instructors or other members of our community, under any circumstances, for reporting concerns about behavior or conduct,” the company added. The response received from DataCamp was not only inadequate, but technologically faulty, as per one of the contractors Noam Ross who pointed out in his blog post that DataCamp had published the blog with a “no-index” tag, meaning it would not show up in aggregated searches like Google results. Thus adding this tag knowingly represents DataCamp’s continued lack of public accountability. OS Keyes said to Business Insider that at this point, the best course of action for DataCamp is a blatant change in leadership. “The investors need to get together and fire the [executive], and follow that by publicly explaining why, apologising, compensating the victim and instituting a much more rigorous set of work expectations,” Keyes said. #Rstats and other data science communities and DataCamp instructors take action One of the contractors Ines Montani expressed this by saying, “I was pretty disappointed, appalled and frustrated by DataCamp's reaction and non-action, especially as more and more details came out about how they essentially tried to sweep this under the rug for almost two years,” Due to their contracts, many instructors cannot take down their DataCamp courses. Instead of removing the courses, many contractors for DataCamp, including Montani, took to Twitter after DataCamp published the blog, urging students to boycott the very courses they designed. https://twitter.com/noamross/status/1116667602741485571 https://twitter.com/daniellequinn88/status/1117860833499832321 https://twitter.com/_tetration_/status/1118987968293875714 Instructors put financial pressures on the company by boycotting their own courses. They also wanted to get the executive responsible for such misbehaviour account for his actions, compensate the victim and compensate those who were fired for complaining—this may ultimately undercut DataCamp’s bottom line. Influential open-source communities, including RStudio, SatRdays, and R-Ladies, have cut all ties with DataCamp to show disappointment with the lack of serious accountability.. CEO steps down “indefinitely” from his role and accepts his mistakes Today Jonathan Cornelissen, accepted his mistake and wrote a public apology for his inappropriate behaviour. He writes, “I want to apologize to a former employee, our employees, and our community. I have failed you twice. First in my behavior and second in my failure to speak clearly and unequivocally to you in a timely manner. I am sorry.” He has also stepped down from his position as the company CEO indefinitely until there is complete review of company’s environment and culture. While it is in the right direction, unfortunately this apology comes to the community very late and is seen as a PR move to appease the backlash from the data science community and other instructors. https://twitter.com/mrsnoms/status/1121235830381645824 9 Data Science Myths Debunked 30 common data science terms explained Why is data science important?
Read more
  • 0
  • 0
  • 4717

article-image-mongodb-is-going-to-acquire-realm-the-mobile-database-management-system-for-39-million
Richard Gall
25 Apr 2019
3 min read
Save for later

MongoDB is going to acquire Realm, the mobile database management system, for $39 million

Richard Gall
25 Apr 2019
3 min read
MongoDB, the open source NoSQL database, is going to acquire mobile database platform Realm. The purchase is certainly one with clear technological and strategic benefits for both companies - and with MongoDB paying $39 million for a company that has up to now raised $40 million since its launch in 2011, it's clear that this is a move that isn't about short term commercial gains. It's important to note that the acquisition is not yet complete. It's expected to close in January 2020 at the end of the second quarter MongoDB's fiscal year. Further details about the acquisition and what it means for both products, will be revealed at MongoDB World in June. Why is MongoDB acquiring Realm? In the materials that announce the launch there's a lot of talk about the alignment between the two projects. "The best thing in the world is when someone just gets you, and you get them" MongoDB CTO Eliot Horowitz wrote in a blog post accompanying the release, "because when you share a vision of the world like that, you can do incredible things together. That’s exactly the case with MongoDB and Realm." At a more fundamental level the acquisition allows MongoDB to do a number of things. It can reach a new community of developers  working primarily in mobile development (according to the press release Realm has 100,000 active users), but it also allows MongoDB to strengthen its capabilities as cloud evolves to become the dominant way that applications are built and hosted. According to Dev Ittycheria, MongoDB President and CEO, Realm "is a natural fit for our global cloud database, MongoDB Atlas, as well as a complement to Stitch, our serverless platform." Serverless might well be a nascent trend at the moment, but the level of conversation and interest around it indicates that it's going to play a big part in application developers lives in the months and years to come. What's in it for Realm? For Realm, the acquisition will give the project access to a new pool of users. With backing from MongoDB, is also provides robust foundations for the project to extend its roadmap and even move faster than it previously would have been able to. Realm CEO David Ratner wrote yesterday (April 24) that: "The combination of MongoDB and Realm will establish the modern standard for mobile application development and data synchronization for a new generation of connected applications and services. MongoDB and Realm are fully committed to investing in the Realm Database and the future of data synchronization, and taking both to the next phase of their evolution. We believe that MongoDB will help accelerate Realm’s product roadmap, go-to-market execution, and support our customers’ use cases at a whole new level of global operational scale." A new chapter for MongoDB? 2019 hasn't been the best year for MongoDB so far. The project withdrew its submission for its controversial Server Side Public License last month following news that Red Hat was dropping it from Enterprise Linux and Fedora. This brought an initiative that the leadership viewed as strategically important in defending MongoDB's interests to a dramatic halt. However, the Realm acquisition sets up a new chapter and could go some way in helping MongoDB bolster itself for a future that it has felt uncertain about.
Read more
  • 0
  • 0
  • 5143

article-image-rust-shares-roadmap-for-2019
Bhagyashree R
24 Apr 2019
3 min read
Save for later

Rust shares roadmap for 2019

Bhagyashree R
24 Apr 2019
3 min read
Yesterday, the Rust team shared its 2019 roadmap, which includes the areas they plan to focus on for the Rust project. This roadmap is based on 2018’s survey results, direct conversations with individual Rust users, and discussions at the 2019 Rust All Hands gathering. The Rust team sees 2019 as a year of “rejuvenation and maturation for the Rust project”. This year, they plan to focus on the following three areas: Improving governance Within the past three years, the Rust project has seen very impressive growth. It now boasts of up to 5000 contributors and over 100 project members. So, those processes that used to serve the small team, has now started to show some cracks. This is why the team plans to revisit these processes. As a part of this focus area, a group called Governance Working Group has been created that will work with each team to help them improve their governance structure. The group will be responsible for examining, documenting, and proposing improvements to the policies and procedures that were used to run the project. Closing out the long-standing requests Last year the Rust team and community shipped many features. Their plan for this year is to step back, assess, and prepare for the future. Instead of coming up with new features, they plan to finish up the initiatives they started. Jonathan Turner, a Rust developer, said, “Let’s finish what we started. As much as is possible and practical this year, let’s set aside new designs and ship what we’ve already designed. Let’s tackle challenges we haven’t had time to give our full attention.” Each sub-team has made a high-level plan for themselves. The Cargo team aims to finish the “almost complete work” on custom registries. The language team plans to ship the in-progress features such as const generics, Generic Associated Types, and specialization. The Library team will be focusing on maintaining the standard library. It also plans to finish up the development of custom allocators associated with instances of collections. Polishing features for better developer experience The third focus area for the Rust team is to “polish” the features that make for great developer experience. In the past few years, Rust has put a lot of effort into foundational work. For instance, they massively refactored the compiler to support incremental compilation and to be better prepared for IDEs. The team plans to improve compile times and IDE support. They plan to work on the documentation and produce “unsafe code guidelines” that explain what unsafe code can and cannot do. The WebAssembly working group will be polishing the wasm support, for example, debugging. To know more, check out the official announcement by Rust. Rust 1.34 releases with alternative cargo registries, stabilized TryFrom and TryInto, and more Chris Dickinson on how to implement Git in Rust Mozilla engineer shares the implications of rewriting browser internals in Rust  
Read more
  • 0
  • 0
  • 4258
article-image-tesla-autonomy-day-takeaways-full-self-driving-computer-robotaxis-launching-next-year-and-more
Bhagyashree R
24 Apr 2019
6 min read
Save for later

Tesla Autonomy Day takeaways: Full Self-Driving computer, Robotaxis launching next year, and more

Bhagyashree R
24 Apr 2019
6 min read
This Monday, Tesla’s “Autonomy Investor Day” kickstarted at its headquarters in Palo Alto. At this invitation-only event, Elon Musk, the CEO of Tesla, with his fellow executives, talked about its new microchip, robotaxis hitting the road by next year, and more. Here are some of the key takeaways from the event: The Full Self-Driving (FSD) computer Tesla shared details of its new custom chip, Full Self-Driving (FSD) computer, previously known as Autopilot Hardware 3.0. Elon Musk, the CEO of Tesla, believes that the FSD computer is “the best chip in the world…objectively.” Tesla replaced Nvidia’s Autopilot 2.5 computer with its own custom chip for Model S and Model X about a month ago. For Model 3 vehicles this change happened about 10 days ago. Musk said, “All cars being produced all have the hardware necessary — computer and otherwise — for full self-driving. All you need to do is improve the software.” FSD is a high-performance, special-purpose chip built by Samsung with main focus on autonomy and safety. It comes with a factor of 21 improvements in frame per second processing as compared to the previous generation Tesla Autopilot hardware, which was powered by Nvidia hardware. The company further shared that retrofits will be offered to current Tesla owners who bought the ‘Full Self-Driving package’ in the next few months. Here’s the new Tesla FSD computer: Credits: Tesla Musk shared that the company has already started working on a next-generation chip. The design of FSD was completed within 2 years of time and Tesla is now about halfway through the design of the next-generation chip. Musk’s claims of building the best chip can be taken with a pinch of salt as it could surely upset some engineers from Nvidia, Mobileye, and other companies who have been in the chip manufacturing market for a long time. Nvidia, in a blog post, along with applauding Tesla for its FSD computer, highlighted “few inaccuracies” in the comparison made by Musk during the event: “It’s not useful to compare the performance of Tesla’s two-chip Full Self Driving computer against NVIDIA’s single-chip driver assistance system. Tesla’s two-chip FSD computer at 144 TOPs would compare against the NVIDIA DRIVE AGX Pegasus computer which runs at 320 TOPS for AI perception, localization and path planning.” While pointing out the “inaccuracies”, Nvidia did miss out the key point here: the power consumption. “Having a system that can do 160 TOPS means little if it uses 500 watts while tesla's 144 TOPS system uses 72 watts,” a Redditor said. Robotaxis will hit the roads in 2020 Musk shared that within the next year or so we will see Tesla’s robotaxis coming into the ride-hailing market giving competition to Uber and Lyft. Musk made a bold claim saying that though, similar to other ride-hailing services, the robotaxis will allow users to hail a Tesla for a ride, they will not have drivers. Musk announced, “I feel very confident predicting that there will be autonomous robotaxis from Tesla next year — not in all jurisdictions because we won’t have regulatory approval everywhere.” He did not share many details on what regulations he was talking about. The service will allow Tesla-owners to add their properly equipped vehicles to Tesla’s own ride-sharing app, following a similar business model as Uber or Airbnb. The company will provide a dedicated number of robotaxis in areas where there are not enough loanable cars. Musk predicted that the average robotaxi will be able to yield $30,000 in gross profit per car, annually. Of this profit, about 25% to 30% will go to Tesla, therefore, an owner will be able to make $21,000 a year. Musk's plans for launching robotaxis next year looks ambitious. Experts and the media are quite skeptical about his plan. The Partners for Automated Vehicle Education (PAVE) industry group tweeted: https://twitter.com/PAVECampaign/status/1120436981220237312 https://twitter.com/RobMcCargow/status/1120961462678245376 Musk says “Anyone relying on lidar is doomed” Musk has been pretty vocal about his dislike towards LIDAR. He calls this technology “a crutch for self-driving cars”. When this topic came up at the event, Musk said: “Lidar is a fool’s errand. Anyone relying on lidar is doomed. Doomed! [They are] expensive sensors that are unnecessary. It’s like having a whole bunch of expensive appendices. Like, one appendix is bad, well now you have a whole bunch of them, it’s ridiculous, you’ll see.” LIDAR, which stands for Light Direction and Ranging, is used by Uber, Waymo, Cruise, and many other self-driving vehicles manufacturing companies. LIDAR projects low-intensity, harmless, and invisible laser beams at a target, or in the case of self-driving cars, all around. The reflected pulses are then measured for return time and wavelength to calculate the distance of an object from the sender. Lidar is capable of producing pretty detailed visualizations of the environment around a self-driving car. However, Tesla believes that this same functionality can be facilitated by cameras. According to Musk, cameras can provide much better resolutions and when combined with the neural net can predict depth very well. Andrej Karpathy, Tesla’s Senior Director of AI, took to the stage to explain the limitations of Lidar. He said, “In that sense, lidar is really a shortcut. It sidesteps the fundamental problems, the important problem of visual recognition, that is necessary for autonomy. It gives a false sense of progress and is ultimately a crutch. It does give, like, really fast demos!”. Karpathy further added, “You were not shooting lasers out of your eyes to get here.” While true, many felt that the reasoning is completely flawed. A Redditor in a discussion thread, said, “Musk's argument that "you drove here using your own two eyes with no lasers coming out of them" is reductive and flawed. It should be obvious to anyone that our eyes are more complex than simple stereo cameras. If the Tesla FSD system can reliably perceive depth at or above the level of the human eye in all conditions, then they have done something truly remarkable. Judging by how Andrej Karpathy deflected the question about how well the system works in snowy conditions, I would assume they have not reached that level.” Check out the live stream of the autonomy day on Tesla’s official website. Tesla v9 to incorporate neural networks for autopilot Tesla is building its own AI hardware for self-driving cars Nvidia Tesla V100 GPUs publicly available in beta on Google Compute Engine and Kubernetes Engine  
Read more
  • 0
  • 0
  • 3606

article-image-openai-five-bots-destroyed-human-dota-2-players-this-weekend
Richard Gall
23 Apr 2019
3 min read
Save for later

OpenAI Five bots destroyed human Dota 2 players this weekend

Richard Gall
23 Apr 2019
3 min read
Last week, the team at OpenAI made it possible for humans to play the OpenAI Five bot at Dota 2 online. The results were staggering - over a period of just a few days, from April 18 to April 21, OpenAI Five had a win rate of 99.4%, winning 7,215 games (that includes humans giving up and abandoning their games 3,140 times) and losing only 42. But perhaps we shouldn't be that surprised. The artificial intelligence bot did, after all, defeat OG, one of the best e-sports teams on the planet earlier this month. https://twitter.com/OpenAI/status/1120421259274334209 What does OpenAI Five's Dota 2 dominance tell us about artificial intelligence? The dominance of OpenAI Five over the weekend is important because it indicates that it is possible to build artificial intelligence that can deal with complex strategic decision-making consistently. Indeed, that's what sets this experiment apart from other artificial intelligence gaming challenges - from the showdown with OG to DeepMind's AlphaZero defeating a professional Go and chess players, bots are typically playing individuals or small teams of players. By taking on the world, it would appear that OpenAI have developed an artificial intelligence system that a large group of intelligent humans with specific domain experience have found it consistently difficult to out-think. Learning how to win The key issue when it comes to artificial intelligence and games - Dota 2 or otherwise - is the ability of the bot to learn. One Dota 2 gamer, quoted on a Reddit thread, said "the bots are locked, they are not learning, but we humans are. We will win." This is true - up to a point. The reality is that they aren't locked - they are, in fact, continually learning, processing the consequences of every decision that is made and feeding it into its system. And although adaptability will remain an issue for any artificial intelligence system, the more games it plays and the more strategies it 'learns' it will essentially build adaptability into its system. This is something OpenAI CTO Greg Brockman noted when responding to suggestions that OpenAI Five's tiny proportion of defeats indicates a lack of adaptability. "When we lost at The International (100% vs pro teams), they said it was because Five can’t do strategy. So we trained for longer. When we lose (0.7% vs the entire Internet), they say it’s because Five can’t adapt." https://twitter.com/gdb/status/1119963994754670594 It's important to remember that this doesn't necessarily signal that much about the possibility of Artificial General Intelligence. OpenAI Five's decision making power is centered around a very specific domain - even if it is one that is relatively complex. However, it does highlight that the relationship between video games and artificial intelligence is particularly important. On the one hand, video games are a space that can help us develop AI further and explore the boundaries of what's possible. But equally, AI will likely evolve the way we think about gaming - and esports - too. Read next: How Artificial Intelligence and Machine Learning can turbocharge a Game Developer’s career
Read more
  • 0
  • 0
  • 4427

article-image-ai-now-institute-publishes-a-report-on-the-diversity-crisis-in-ai-and-offers-12-solutions-to-fix-it
Bhagyashree R
22 Apr 2019
7 min read
Save for later

AI Now Institute publishes a report on the diversity crisis in AI and offers 12 solutions to fix it

Bhagyashree R
22 Apr 2019
7 min read
Earlier this month, the AI Now Institute published a report, authored by Sarah Myers West, Meredith Whittaker, and Kate Crawford, highlighting the link between the diversity issue in the current AI industry and the discriminating behavior of AI systems. The report further recommends some solutions to these problems that companies and the researchers behind these systems need to adopt to address these issues. Sarah Myers West is a postdoc researcher at the AI Now Institute and an affiliate researcher at the Berkman-Klein Center for Internet and Society. Meredith Whittaker is the co-founder of the AI Now Institute and leads Google's Open Research Group and the Google Measurement Lab.  Kate Crawford is a Principal Researcher at Microsoft Research and the co-founder and Director of Research at the AI Now Institute. Kate Crawford tweeted about this study. https://twitter.com/katecrawford/status/1118509988392112128 The AI industry lacks diversity, gender neutrality, and bias-free systems In recent years, we have come across several cases of “discriminating systems”. Facial recognition systems miscategorize black people and sometimes fails to work for trans drivers. When trained in online discourse, chatbots easily learn racist and misogynistic language. This type of behavior by machines is actually a reflection of society. “In most cases, such bias mirrors and replicates existing structures of inequality in the society,” says the report. The study also sheds light on gender bias in the current workforce. According to the report, only 18% of authors at some of the biggest AI conferences are women. On the other side of the spectrum are men who cover 80%. The tech giants, Facebook and Google, have a meager 15% and 10% women as their AI research staff. The situation for black workers in the AI industry looks even worse. While Facebook and Microsoft have 4% of their current workforce as black workers, Google stands at just 2.5%. Also, vast majority of AI studies assume gender is binary, and commonly assigns people as ‘male’ or ‘female’ based on physical appearance and stereotypical assumptions, erasing all other forms of gender identity. The report further reveals that, though there have been various “pipeline studies” to check the flow of diverse job candidates, they have failed to show substantial progress in bringing diversity in the AI industry. “The focus on the pipeline has not addressed deeper issues with workplace cultures, power asymmetries, harassment, exclusionary hiring practices, unfair compensation, and tokenization that are causing people to leave or avoid working in the AI sector altogether,” the report reads. What steps can industries take to address bias and discrimination in AI Systems The report lists 12 recommendations that AI researchers and companies should employ to improve workplace diversity and address bias and discrimination in AI systems. Publish compensation levels, including bonuses and equity, across all roles and job categories, broken down by race and gender. End pay and opportunity inequality, and set pay and benefit equity goals that include contract workers, temps, and vendors. Publish harassment and discrimination transparency reports, including the number of claims over time, the types of claims submitted, and actions taken. Change hiring practices to maximize diversity: include targeted recruitment beyond elite universities, ensure more equitable focus on under-represented groups, and create more pathways for contractors, temps, and vendors to become full-time employees. Commit to transparency around hiring practices, especially regarding how candidates are leveled, compensated, and promoted. Increase the number of people of color, women and other under-represented groups at senior leadership levels of AI companies across all departments. Ensure executive incentive structures are tied to increases in hiring and retention of underrepresented groups. For academic workplaces, ensure greater diversity in all spaces where AI research is conducted, including AI-related departments and conference committees. Remedying bias in AI systems is almost impossible when these systems are opaque. Transparency is essential, and begins with tracking and publicizing where AI systems are used, and for what purpose. Rigorous testing should be required across the lifecycle of AI systems in sensitive domains. Pre-release trials, independent auditing, and ongoing monitoring are necessary to test for bias, discrimination, and other harms. The field of research on bias and fairness needs to go beyond technical debiasing to include a wider social analysis of how AI is used in context. This necessitates including a wider range of disciplinary expertise. The methods for addressing bias and discrimination in AI need to expand to include assessments of whether certain systems should be designed at all, based on a thorough risk assessment. AI-related departments and conference committees. Credits: AI Now Institute Bringing diversity in the AI workforce In order to address the diversity issue in the AI industry, companies need to make changes in the current hiring practices. They should have a more equitable focus on under-represented groups. People of color, women, and other under-represented groups should get fair chance to get into senior leadership level of AI companies across all departments. Further opportunities should be created for contractors, temps, and vendors to become full-time employees. To bridge the gender pay gap in the AI industry, it is important that companies maintain transparency regarding the compensation levels, including bonuses and equity, regardless of gender or race. In the past few years, several cases of sexual misconducts involving some of the biggest companies like Google, Microsoft, have come into light because of movements like #MeToo, Google Walkout, and more. These movements gave the victims and other supporting employees  the courage to speak against employees at higher positions who were taking undue advantage of their power. There are cases were the sexual harassment complaints were not taken seriously by the HRs and victims were told to just “get over it”. This is why, companies should  publish harassment and discrimination transparency reports that include information like the number and types of claims made and the actions taken by the company. Academic workplaces should ensure diversity in all AI-related departments and conference committees. In the past, some of the biggest AI conferences like Neural Information Processing Systems conference has failed to provide a welcoming and safer environment for women. In a survey conducted last year, many respondents shared that they have experienced sexual harassment. Women reported persistent advances from men at the conference. The organizers of such conferences should ensure an inclusive and welcoming environment for everyone. Addressing bias and discrimination in AI systems To address bias and discrimination in AI systems, the report recommends to do rigorous testing across the lifecycle of these systems. These systems should have pre-release trials, independent auditing, and monitoring to check bias, discrimination, and other harms. Looking at the social implications of AI systems, just addressing the algorithmic bias is not enough. “The field of research on bias and fairness needs to go beyond technical debiasing to include a wider social analysis of how AI is used in context. This necessitates including a wider range of disciplinary expertise,” says the report. While assessing a AI system, researchers and developers should also check whether designing a certain system is required at all, considering the risks it poses. The study calls for re-evaluating the current AI systems used for classifying, detecting, and predicting the race and gender. The idea of identifying a race or gender just by appearance is flawed and can be easily abused. Especially, systems that use physical appearance to find interior states, for instance, those that claim to detect sexuality from headshots. These systems are urgently in need to be checked. To know more in detail, read the full report: Discriminating Systems. Microsoft’s #MeToo reckoning: female employees speak out against workplace harassment and discrimination Desmond U. Patton, Director of SAFElab shares why AI systems should be a product of interdisciplinary research and diverse teams Google’s Chief Diversity Officer, Danielle Brown resigns to join HR tech firm Gusto
Read more
  • 0
  • 0
  • 4462
article-image-how-to-build-topic-models-in-r-tutorial
Natasha Mathur
22 Apr 2019
8 min read
Save for later

How to build topic models in R [Tutorial]

Natasha Mathur
22 Apr 2019
8 min read
Topic models are a powerful method to group documents by their main topics. Topic models allow probabilistic modeling of term frequency occurrence in documents. The fitted model can be used to estimate the similarity between documents, as well as between a set of specified keywords using an additional layer of latent variables, which are referred to as topics (Grun and Hornik, 2011). In essence, a document is assigned to a topic based on the distribution of the words in that document, and the other documents in that topic will have roughly the same frequency of words. In this tutorial, we will look at a useful framework for text mining, called topic models. We will apply the framework to the State of the Union addresses. In building topic models, the number of topics must be determined before running the algorithm (k-dimensions). If no prior reason for the number of topics exists, then you can build several and apply judgment and knowledge to the final selection. There are different methods that come under Topic Modeling. We'll look at LDA with Gibbs sampling. This method is quite complicated mathematically, but my intent is to provide an introduction so that you are at least able to describe how the algorithm learns to assign a document to a topic in layperson terms. If you are interested in mastering the math associated with the method, block out a couple of hours on your calendar and have a go at it. Excellent background material can be found here.  This tutorial is an excerpt taken from the book 'Mastering Machine Learning with R - Third Edition' written by Cory Lesmeister. The book explores expert techniques for solving data analytics and covers machine learning challenges that can help you gain insights from complex projects and power up your applications. Talking about LDA  or Latent Dirichlet Allocation in topic modeling, it is a generative process, and works in the following manner to iterate to a steady state: For each document (j), there are 1 to j documents. We will randomly assign a multinomial distribution (Dirichlet distribution) to the topics (k) with 1 to k topics, for example, document A is 25 percent topic one, 25 percent topic two, and 50 percent topic three. Probabilistically, for each word (i), there are 1 to i words to a topic (k); for example, the word mean has a probability of 0.25 for the topic statistics. For each word (i) in document (j) and topic (k), calculate the proportion of words in that document assigned to that topic; note it as the probability of topic (k) with document (j), p(k|j), and the proportion of word (i) in topic (k) from all the documents containing the word. Note it as the probability of word (i) with topic (k), p(i|k). Resample, that is, assign w a new t based on the probability that t contains w, which is based on p(k|j) times p(i|k). Rinse and repeat; over numerous iterations, the algorithm finally converges and a document is assigned a topic based on the proportion of words assigned to a topic in that document. The LDA assumes that the order of words and documents does not matter. There has been work done to relax these assumptions in order to build models of language generation and sequence models over time (known as dynamic topic modeling or DTM). Applying Topic models in State of the Union addresses We will leave behind the 19th century and look at these recent times of trial and tribulation (1965 through 2016). On looking at this data, I found something interesting and troubling. Let's take a look at the 1970s: > sotu_meta[185:191, 1:4] # A tibble: 7 x 4 president year years_active party <chr> <int> <chr> <chr> 1 Richard M. Nixon 1970 1969-1973 Republican 2 Richard M. Nixon 1971 1969-1973 Republican 3 Richard M. Nixon 1972 1969-1973 Republican 4 Richard M. Nixon 1972 1969-1973 Republican 5 Richard M. Nixon 1974 1973-1974 Republican 6 Richard M. Nixon 1974 1973-1974 Republican 7 Gerald R. Ford 1975 1974-1977 Republican We see there are two 1972 and two 1974 addresses, but none for 1973. What? I went to the Nixon Foundation website, spent about 10 minutes trying to deconflict this, and finally threw my hands in the air and decided on implementing a quick fix. Be advised that there are a number of these conflicts to put in order: > sotu_meta[188, 2] <- "1972_2" > sotu_meta[190, 2] <- "1974_2" > sotu_meta[157, 2] <- "1945_2" > sotu_meta[166, 2] <- "1953_2" > sotu_meta[170, 2] <- "1956_2" > sotu_meta[176, 2] <- "1961_2" > sotu_meta[195, 2] <- "1978_2" > sotu_meta[197, 2] <- "1979_2" > sotu_meta[199, 2] <- "1980_2" > sotu_meta[201, 2] <- "1981_2" An email to the author of this package is in order. I won't bother with that, but feel free to solve the issue yourself. With this tragedy behind us, we'll go through tokenizing and removing stop words again for our relevant time frame: > sotu_meta_recent <- sotu_meta %>% dplyr::filter(year > 1964) > sotu_meta_recent %>% tidytext::unnest_tokens(word, text) -> sotu_unnest_recent > sotu_recent <- sotu_unnest_recent %>% dplyr::anti_join(stop_words, by = "word") As discussed previously, we need to put the data into a DTM before building a model. This is done by creating a word count grouped by year, then passing that to the cast_dtm() function: > sotu_recent %>% dplyr::group_by(year) %>% dplyr::count(word) -> lda_words > sotu_dtm <- tidytext::cast_dtm(lda_words, year, word, n) Let's get our model built. I'm going to create six different topics using the Gibbs method, and I specified verbose. It should run 2,000 iterations: > sotu_lda <- topicmodels::LDA( sotu_dtm, k = 6, method = "Gibbs", control = list(seed = 1965, verbose = 1) ) > sotu_lda A LDA_Gibbs topic model with 6 topics. The algorithm gives each topic a number. We can see what year is mapped to what topic. I abbreviate the output since 2002: > topicmodels::topics(sotu_lda) 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 We see a clear transition between Bush and Obama from topic 2 to topic 4. Here is a table of the count of topics: > table(topicmodels::topics(sotu_lda)) 1 2 3 4 5 6 8 7 5 18 14 5 Topic 4 is the most prevalent, which is associated with Clinton's term also. This output gives us the top five words associated with each topic: > topicmodels::terms(sotu_lda, 5) Topic 1 Topic 2 Topic 3 [1,] "future" "america" "administration" [2,] "tax" "security" "congress" [3,] "spending" "country" "economic" [4,] "government" "world" "legislation" [5,] "economic" "iraq" "energy" Topic 4 Topic 5 Topic 6 [1,] "people" "world" "federal" [2,] "american" "people" "programs" [3,] "jobs" "american" "government" [4,] "america" "congress" "program" [5,] "children" "peace" "act" This all makes good sense, and topic 2 is spot on for the time. If you drill down further to, say, 10, 15, or 20 words, it is even more revealing, but I won't bore you further. What about an application in the tidy ecosystem and a visualization? Certainly! We'll turn the model object into a data frame first and in the process capture the per-topic-per-word probabilities called beta: > lda_topics <- tidytext::tidy(sotu_lda, matrix = "beta") > ap_top_terms <- lda_topics %>% dplyr::group_by(topic) %>% dplyr::top_n(10, beta) %>% dplyr::ungroup() %>% dplyr::arrange(topic, -beta) We can explore that data further or just plot it as follows: > ap_top_terms %>% dplyr::mutate(term = reorder(term, beta)) %>% ggplot2::ggplot(ggplot2::aes(term, beta, fill = factor(topic))) + ggplot2::geom_col(show.legend = FALSE) + ggplot2::facet_wrap(~ topic, scales = "free") + ggplot2::coord_flip() + ggthemes::theme_economist_white() The output of the preceding code is as follows: This is the top 10 words per topic based on the beta probability. Another thing we can do is look at the probability an address is related to a topic. This is referred to as gamma in the model and we can pull those in just like the beta: > ap_documents <- tidytext::tidy(sotu_lda, matrix = "gamma") We now have the probabilities of an address per topic. Let's look at the 1981 Ronald Reagan values: > dplyr::filter(ap_documents, document == "1981") # A tibble: 6 x 3 document topic gamma <chr> <int> <dbl> 1 1981 1 0.286 2 1981 2 0.0163 3 1981 3 0.0923 4 1981 4 0.118 5 1981 5 0.0777 6 1981 6 0.411 Topic 1 is a close second in the topic race. If you think about it, this means that more than six topics would help to create better separation in the probabilities. However, I like just six topics for this tutorial for the purpose of demonstration. In this tutorial, we looked at topic models in R. We applied the framework to the State of the Union addresses.  If you want to stay updated with expert techniques for solving data analytics and explore other machine learning challenges in R, be sure to check out the book 'Mastering Machine Learning with R - Third Edition'. How to make machine learning based recommendations using Julia [Tutorial] The rise of machine learning in the investment industry GitHub Octoverse: top machine learning packages, languages, and projects of 2018
Read more
  • 0
  • 0
  • 8866

article-image-brett-lantz-shows-how-data-scientists-learn-building-algorithms-in-third-edition-machine-learning-r
Packt Editorial Staff
22 Apr 2019
3 min read
Save for later

The hands-on guide to Machine Learning with R by Brett Lantz

Packt Editorial Staff
22 Apr 2019
3 min read
If science fiction stories are to be believed, the invention of Artificial Intelligence inevitably leads to apocalyptic wars between machines and their makers. Thankfully, at the time of this writing, machines still require user input. Though your impressions of Machine Learning may be colored by these mass-media depictions, today's algorithms are too application-specific to pose any danger of becoming self-aware. The goal of today's Machine Learning is not to create an artificial brain, but rather to assist us with making sense of the world's massive data stores. Conceptually, the learning process involves the abstraction of data into a structured representation, and the generalization of the structure into action that can be evaluated for utility. In practical terms, a machine learner uses data containing examples and features of the concept to be learned, then summarizes this data in the form of a model, which is used for predictive or descriptive purposes. The field of machine learning provides a set of algorithms that transform data into actionable knowledge. Among the many possible methods, machine learning algorithms are chosen on the basis of the input data and the learning task. This fact makes machine learning well-suited to the present-day era of big data. Machine Learning with R, Third Edition introduces you to the fundamental concepts that define and differentiate the most commonly used machine learning approaches and how easy it is to use R to start applying machine learning to real-world problems. Many of the algorithms needed for machine learning are not included as part of the base R installation. Instead, the algorithms are available via a large community of experts who have shared their work freely. These powerful tools are available to download at no cost, but must be installed on top of base R manually. This book covers a small portion of all of R's machine learning packages and will get you up to speed with the learning landscape of machine learning with R. Machine Learning with R, Third Edition updates the classic R data science book with newer and better libraries, advice on ethical and bias issues in machine learning, and an introduction to deep learning. Whether you are an experienced R user or new to the language, Brett Lantz teaches you everything you need to uncover key insights, make new predictions, and visualize your findings. Introduction to Machine Learning with R Machine Learning with R How to make machine learning based recommendations using Julia [Tutorial]
Read more
  • 0
  • 0
  • 4478