Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon

How-To Tutorials

7019 Articles
article-image-stack-overflow-confirms-production-systems-hacked
Vincy Davis
17 May 2019
2 min read
Save for later

Stack Overflow confirms production systems hacked

Vincy Davis
17 May 2019
2 min read
Almost after a week of the attack, Stack Overflow admitted in an official security update yesterday, that their production systems has been hacked. “Over the weekend, there was an attack on Stack Overflow. We have confirmed that some level of production access was gained on May 11”, said Mary Ferguson ,VP of Engineering at Stack Overflow. In this short update, the company has mentioned that they are investigating the extent of the access and are addressing all the known vulnerabilities. Though not confirmed, the company has identified no breach of customer or user data. https://twitter.com/gcluley/status/1129260135778607104 Some users are acknowledging the fact that that the firm has at least come forward and accepted the security violation. A user on Reddit said, “Wow. I'm glad they're letting us know early, but this sucks” There are other users who think that security breach due to hacking is very common nowadays. A user on Hacker News commented, “I think we've reached a point where it's safe to say that if you're using a service -any service - assume your data is breached (or willingly given) and accessible to some unknown third party. That third party can be the government, it can be some random marketer or it can be a malicious hacker. Just hope that you have nothing anywhere that may be of interest or value to anyone, anywhere. Good luck.” Few days ago, there were reports that Stack Overflow directly links to Facebook profile pictures. This means that the linking unintentionally allows user activity throughout Stack Exchange to be tracked by Facebook and also tracks the topics that the users are interested in. Read More: Facebook again, caught tracking Stack Overflow user activity and data Stack Overflow has also assured users that more information will be provided to them, once the company concludes the investigation. Stack Overflow survey data further confirms Python’s popularity as it moves above Java in the most used programming language list 2019 Stack Overflow survey: A quick overview Stack Overflow is looking for a new CEO as Joel Spolsky becomes Chairman
Read more
  • 0
  • 0
  • 2819

article-image-8-tech-companies-and-18-governments-sign-the-christchurch-call-to-curb-online-extremism-the-us-backs-off
Sugandha Lahoti
16 May 2019
6 min read
Save for later

8 tech companies and 18 governments sign the Christchurch Call to curb online extremism; the US backs off

Sugandha Lahoti
16 May 2019
6 min read
Yesterday, New Zealand Prime Minister, Jacinda Ardern, and French President, Emmanuel Macron brought together world governments and leaders from the tech sector to adopt the Christchurch Call. This call is a non-binding agreement to eliminate terrorist and violent extremist content online after the Christ church terror attack, which took place March 15 and killed 51 Muslim worshipers in Christchurch mosques in New Zealand. Tech companies scrambled to take action due to the speed and volume of content which was live streamed on Facebook and then subsequently uploaded, reuploaded and shared by the users worldwide. “The Christchurch Call is a commitment by Governments and tech companies to eliminate terrorist and violent extremist content online. It rests on the conviction that a free, open and secure internet offers extraordinary benefits to society. Respect for freedom of expression is fundamental. However, no one has the right to create and share terrorist and violent extremist content online.”, reads the call. These tech companies and governments will also be developing tools to prevent the upload of terrorist and violent extremist content; countering the roots of violent extremism; increasing transparency around the removal and detection of content, and reviewing how companies’ algorithms to direct users to violent extremist content. Representatives from Facebook, Google, Microsoft, Twitter, and other tech giants have also agreed to identify "appropriate checks on live streaming, aimed at reducing the risk of disseminating terrorist and violent extremist content online". Two days ago in a statement, Facebook declared that from now on they will start restricting users from using Facebook Live if they break certain rules-including their Dangerous Organizations and Individuals policy. In addition to signing the Christchurch Call, Amazon, Facebook, Google, Twitter, and Microsoft are publishing nine steps that they will take to address the abuse of technology to spread terrorist and violent extremist content. These nine steps include five individual actions that each company is committing to take and a further four collaborative actions they’ll take together. Terms of Use user reporting of terrorist and violent extremist content; enhancing technology; live streaming, and transparency reports. Also included were four collaborative actions such as shared technology development, crisis protocols, educating the public about terrorist and extremist violent content online and combating hate and bigotry. Also, to be noted, Twitter CEO Jack Dorsey was the only social media CEO to attend the Call. Facebook’s Mark Zuckerberg was a no show with Facebook’s head of global affairs Nick Clegg attending the call. Microsoft was represented by President Brad Smith while Wikimedia was represented by Wikipedia founder Jimmy Wales. Senior Vice President for Global Affairs Kent Walker participated on behalf of Google. https://twitter.com/Policy/status/1128693729181749248 The signing of the Christchurch Call was organized around a meeting of digital ministers from the Group of 7 nations this week in Paris. Until now 18 countries and tech organizations have joined the call, pledging to counter the drivers of terrorism and violent extremism, and ensure effective enforcement of applicable laws. The Call was adopted at the meeting by France, New Zealand, Canada, Indonesia, Ireland, Jordan, Norway, Senegal, the UK, and the European Commission. Other countries who have adopted the Call but were not at the meeting are Australia, Germany, India, Italy, Japan, the Netherlands, Spain, and Sweden. Sri Lanka has not signed the agreement. In April, Sri Lanka has massive bombings on churches and hotels, which killed 321 people and led to the Sri Lankan government issuing a blanket social media ban in the weeks to come for fear of them inciting further violence, fear and spread of misinformation. Sri Lanka’s junior minister for defense, Ruwan Wijewardene, in an initial investigation had revealed the bombings were related to New Zealand mosque attack. However, Ardern denied those claims stating that the New Zealand government was not aware of any such intelligence. Interestingly, the U.S also refused to join other nations to pledge the Christchurch Call. A statement from the White House reads, “While the United States is not currently in a position to join the endorsement, we continue to support the overall goals reflected in the Call. We will continue to engage governments, industry, and civil society to counter terrorist content on the Internet,” “We continue to be proactive in our efforts to counter terrorist content online while also continuing to respect freedom of expression and freedom of the press,” the statement reads. “Further, we maintain that the best tool to defeat terrorist speech is productive speech, and thus we emphasize the importance of promoting credible, alternative narratives as the primary means by which we can defeat terrorist messaging.” Critics shot down White House’s refusal to pledge to the non-binding Christchurch Call. https://twitter.com/cwarzel/status/1128701486144282624 https://twitter.com/MarkMellman/status/1128691586001326082 https://twitter.com/StollmeyerEU/status/1128782543585775616 https://twitter.com/JamilSmith/status/1128742138903056385 The Christchurch call is the first time when governments and tech companies have jointly agreed to a set of commitments and ongoing collaboration to make the internet safer. “We can be proud of what we have started with the adoption of the Christchurch Call. We’ve taken practical steps to try and stop what we experienced in Christchurch from happening again” Jacinda Ardern said. “From here, I will work alongside others signed up to the Christchurch Call to bring more partners on board, and develop a range of practical initiatives to ensure the pledge we have made today is delivered”, she added. Digital ministers of the Group of 7 nations, who signed the Christ Church call are also meeting today to discuss an upcoming charter that would cover broader territory than the Christchurch Call on toxic content and tech regulation in general. https://twitter.com/LeoVaradkar/status/1128771650391093250 https://twitter.com/adamihad/status/1128298612734201858 Although critics praised the call for its use of existing laws against extremist and terrorist content and its insistence that legal and platform content regulation measures comply with intelligent human rights law, they also listed certain shortcomings. https://twitter.com/mediamorphis/status/1128773160357302273 Read the full text of Christchurch call here. Google and Facebook working hard to clean image after the media backlash from the Christchurch terrorist attack Facebook tightens rules around live streaming in response to the Christchurch terror attack How social media enabled and amplified the Christchurch terrorist attack
Read more
  • 0
  • 0
  • 2252

article-image-implementing-autocompletion-in-a-react-material-ui-application-tutorial
Bhagyashree R
16 May 2019
14 min read
Save for later

Implementing autocompletion in a React Material UI application [Tutorial]

Bhagyashree R
16 May 2019
14 min read
Web applications typically provide autocomplete input fields when there are too many choices to select from. Autocomplete fields are like text input fields—as users start typing, they are given a smaller list of choices based on what they've typed. Once the user is ready to make a selection, the actual input is filled with components called Chips—especially relevant when the user needs to be able to make multiple selections. In this article, we will start by building an Autocomplete component. Then we will move on to implementing multi-value selection and see how to better serve the autocomplete data through an API. To help our users better understand the results we will also implement a feature that highlights the matched portion of the string value. This article is taken from the book React Material-UI Cookbook by Adam Boduch. This book will serve as your ultimate guide to building compelling user interfaces with React and Material Design. To follow along with the examples implemented in this article, you can download the code from the book’s GitHub repository. Building an Autocomplete component Material-UI doesn't actually come with an Autocomplete component. The reason is that, since there are so many different implementations of autocomplete selection components in the React ecosystem already, it doesn't make sense to provide another one. Instead, you can pick an existing implementation and augment it with Material-UI components so that it can integrate nicely with your Material-UI application. How to do it? You can use the Select component from the react-select package to provide the autocomplete functionality that you need. You can use Select properties to replace key autocomplete components with Material-UI components so that the autocomplete matches the look and feel of the rest of your app. Let's make a reusable Autocomplete component. The Select component allows you to replace certain aspects of the autocomplete experience. In particular, the following are the components that you'll be replacing: Control: The text input component to use Menu: A menu with suggestions, displayed when the user starts typing NoOptionsMessage: The message that's displayed when there aren't any suggestions to display Option: The component used for each suggestion in Menu Placeholder: The placeholder text component for the text input SingleValue: The component for showing a value once it's selected ValueContainer: The component that wraps SingleValue IndicatorSeparator: Separates buttons on the right side of the autocomplete ClearIndicator: The component used for the button that clears the current value DropdownIndicator: The component used for the button that shows Menu Each of these components is replaced with Material-UI components that change the look and feel of the autocomplete. Moreover, you'll have all of this as new Autocomplete components that you can reuse throughout your app. Let's look at the result before diving into the implementation of each replacement component. Following is what you'll see when the screen first loads: If you click on the down arrow, you'll see a menu with all the values, as follows: Try typing tor into the autocomplete text field, as follows: If you make a selection, the menu is closed and the text field is populated with the selected value, as follows: You can change your selection by opening the menu and selecting another value, or you can clear the selection by clicking on the clear button to the right of the text. How does it work? Let's break down the source by looking at the individual components that make up the Autocomplete component and replacing pieces of the Select component. Then, we'll look at the final Autocomplete component. Text input control Here's the source for the Control component: const inputComponent = ({ inputRef, ...props }) => ( <div ref={inputRef} {...props} /> ); const Control = props => ( <TextField fullWidth InputProps={{ inputComponent, inputProps: { className: props.selectProps.classes.input, inputRef: props.innerRef, children: props.children, ...props.innerProps } }} {...props.selectProps.textFieldProps} /> ); The inputComponent() function is a component that passes the inputRef value—a reference to the underlying input element—to the ref prop. Then, inputComponent is passed to InputProps to set the input component used by TextField. This component is a little bit confusing because it's passing references around and it uses a helper component for this purpose. The important thing to remember is that the job of Control is to set up the Select component to use a Material-UITextField component. Options menu Here's the component that displays the autocomplete options when the user starts typing or clicks on the down arrow: const Menu = props => ( <Paper square className={props.selectProps.classes.paper} {...props.innerProps} > {props.children} </Paper> ); The Menu component renders a Material-UI Paper component so that the element surrounding the options is themed accordingly. No options available Here's the NoOptionsMessage component. It is rendered when there aren't any autocomplete options to display, as follows: const NoOptionsMessage = props => ( <Typography color="textSecondary" className={props.selectProps.classes.noOptionsMessage} {...props.innerProps} > {props.children} </Typography> ); This renders a Typography component with textSecondary as the color property value. Individual option Individual options that are displayed in the autocomplete menu are rendered using the MenuItem component, as follows: const Option = props => ( <MenuItem buttonRef={props.innerRef} selected={props.isFocused} component="div" style={{ fontWeight: props.isSelected ? 500 : 400 }} {...props.innerProps} > {props.children} </MenuItem> ); The selected and style properties alter the way that the item is displayed, based on the isSelected and isFocused properties. The children property sets the value of the item. Placeholder text The Placeholder text of the Autocomplete component is shown before the user types anything or makes a selection, as follows: const Placeholder = props => ( <Typography color="textSecondary" className={props.selectProps.classes.placeholder} {...props.innerProps} > {props.children} </Typography> ); The Material-UI Typography component is used to theme the Placeholder text. SingleValue Once again, the Material-UI Typography component is used to render the selected value from the menu within the autocomplete input, as follows: const SingleValue = props => ( <Typography className={props.selectProps.classes.singleValue} {...props.innerProps} > {props.children} </Typography> ); ValueContainer The ValueContainer component is used to wrap the SingleValue component with a div and the valueContainer CSS class, as follows: const ValueContainer = props => ( <div className={props.selectProps.classes.valueContainer}> {props.children} </div> ); IndicatorSeparator By default, the Select component uses a pipe character as a separator between the buttons on the right side of the autocomplete menu. Since they're going to be replaced by Material-UI button components, this separator is no longer necessary, as follows: const IndicatorSeparator = () => null; By having the component return null, nothing is rendered. Clear option indicator This button is used to clear any selection made previously by the user, as follows: const ClearIndicator = props => ( <IconButton {...props.innerProps}> <CancelIcon /> </IconButton> ); The purpose of this component is to use the Material-UI IconButton component and to render a Material-UI icon. The click handler is passed in through innerProps. Show menu indicator Just like the ClearIndicator component, the DropdownIndicator component replaces the button used to show the autocomplete menu with an icon from Material-UI, as follows: const DropdownIndicator = props => ( <IconButton {...props.innerProps}> <ArrowDropDownIcon /> </IconButton> ); Styles Here are the styles used by the various sub-components of the autocomplete: const useStyles = makeStyles(theme => ({ root: { flexGrow: 1, height: 250 }, input: { display: 'flex', padding: 0 }, valueContainer: { display: 'flex', flexWrap: 'wrap', flex: 1, alignItems: 'center', overflow: 'hidden' }, noOptionsMessage: { padding: `${theme.spacing(1)}px ${theme.spacing(2)}px` }, singleValue: { fontSize: 16 }, placeholder: { position: 'absolute', left: 2, fontSize: 16 }, paper: { position: 'absolute', zIndex: 1, marginTop: theme.spacing(1), left: 0, right: 0 } })); The Autocomplete Finally, following is the Autocomplete component that you can reuse throughout your application: Autocomplete.defaultProps = { isClearable: true, components: { Control, Menu, NoOptionsMessage, Option, Placeholder, SingleValue, ValueContainer, IndicatorSeparator, ClearIndicator, DropdownIndicator }, options: [ { label: 'Boston Bruins', value: 'BOS' }, { label: 'Buffalo Sabres', value: 'BUF' }, { label: 'Detroit Red Wings', value: 'DET' }, { label: 'Florida Panthers', value: 'FLA' }, { label: 'Montreal Canadiens', value: 'MTL' }, { label: 'Ottawa Senators', value: 'OTT' }, { label: 'Tampa Bay Lightning', value: 'TBL' }, { label: 'Toronto Maple Leafs', value: 'TOR' }, { label: 'Carolina Hurricanes', value: 'CAR' }, { label: 'Columbus Blue Jackets', value: 'CBJ' }, { label: 'New Jersey Devils', value: 'NJD' }, { label: 'New York Islanders', value: 'NYI' }, { label: 'New York Rangers', value: 'NYR' }, { label: 'Philadelphia Flyers', value: 'PHI' }, { label: 'Pittsburgh Penguins', value: 'PIT' }, { label: 'Washington Capitals', value: 'WSH' }, { label: 'Chicago Blackhawks', value: 'CHI' }, { label: 'Colorado Avalanche', value: 'COL' }, { label: 'Dallas Stars', value: 'DAL' }, { label: 'Minnesota Wild', value: 'MIN' }, { label: 'Nashville Predators', value: 'NSH' }, { label: 'St. Louis Blues', value: 'STL' }, { label: 'Winnipeg Jets', value: 'WPG' }, { label: 'Anaheim Ducks', value: 'ANA' }, { label: 'Arizona Coyotes', value: 'ARI' }, { label: 'Calgary Flames', value: 'CGY' }, { label: 'Edmonton Oilers', value: 'EDM' }, { label: 'Los Angeles Kings', value: 'LAK' }, { label: 'San Jose Sharks', value: 'SJS' }, { label: 'Vancouver Canucks', value: 'VAN' }, { label: 'Vegas Golden Knights', value: 'VGK' } ] }; The piece that ties all of the previous components together is the components property that's passed to Select. This is actually set as a default property in Autocomplete, so it can be further overridden. The value passed to components is a simple object that maps the component name to its implementation. Selecting autocomplete suggestions In the previous section, you built an Autocomplete component capable of selecting a single value. Sometimes, you need the ability to select multiple values from an Autocomplete component. The good news is that, with a few small additions, the component that you created in the previous section already does most of the work. How to do it? Let's walk through the additions that need to be made in order to support multi-value selection in the Autocomplete component, starting with the new MultiValue component, as follows: const MultiValue = props => ( <Chip tabIndex={-1} label={props.children} className={clsx(props.selectProps.classes.chip, { [props.selectProps.classes.chipFocused]: props.isFocused })} onDelete={props.removeProps.onClick} deleteIcon={<CancelIcon {...props.removeProps} />} /> ); The MultiValue component uses the Material-UI Chip component to render a selected value. In order to pass MultiValue to Select, add it to the components object that's passed to Select: components: { Control, Menu, NoOptionsMessage, Option, Placeholder, SingleValue, MultiValue, ValueContainer, IndicatorSeparator, ClearIndicator, DropdownIndicator }, Now you can use your Autocomplete component for single value selection, or for multi-value selection. You can add the isMulti property with a default value of true to defaultProps, as follows: isMulti: true, Now, you should be able to select multiple values from the autocomplete. How does it work? Nothing looks different about the autocomplete when it's first rendered, or when you show the menu. When you make a selection, the Chip component is used to display the value. Chips are ideal for displaying small pieces of information like this. Furthermore, the close button integrates nicely with it, making it easy for the user to remove individual selections after they've been made. Here's what the autocomplete looks like after multiple selections have been made: API-driven Autocomplete You can't always have your autocomplete data ready to render on the initial page load. Imagine trying to load hundreds or thousands of items before the user can interact with anything. The better approach is to keep the data on the server and supply an API endpoint with the autocomplete text as the user types. Then you only need to load a smaller set of data returned by the API. How to do it? Let's rework the example from the previous section. We'll keep all of the same autocomplete functionality, except that, instead of passing an array to the options property, we'll pass in an API function that returns a Promise. Here's the API function that mocks an API call that resolves a Promise: const someAPI = searchText => new Promise(resolve => { setTimeout(() => { const teams = [ { label: 'Boston Bruins', value: 'BOS' }, { label: 'Buffalo Sabres', value: 'BUF' }, { label: 'Detroit Red Wings', value: 'DET' }, ... ]; resolve( teams.filter( team => searchText && team.label .toLowerCase() .includes(searchText.toLowerCase()) ) ); }, 1000); }); This function takes a search string argument and returns a Promise. The same data that would otherwise be passed to the Select component in the options property is filtered here instead. Think of anything that happens in this function as happening behind an API in a real app. The returned Promise is then resolved with an array of matching items following a simulated latency of one second. You also need to add a couple of components to the composition of the Select component (we're up to 13 now!), as follows: const LoadingIndicator = () => <CircularProgress size={20} />; const LoadingMessage = props => ( <Typography color="textSecondary" className={props.selectProps.classes.noOptionsMessage} {...props.innerProps} > {props.children} </Typography> ); The LoadingIndicator component is shown on the right the autocomplete text input. It's using the CircularProgress component from Material-UI to indicate that the autocomplete is doing something. The LoadingMessage component follows the same pattern as the other text replacement components used with Select in this example. The loading text is displayed when the menu is shown, but the Promise that resolves the options is still pending. Lastly, there's the Select component. Instead of using Select, you need to use the AsyncSelect version, as follows: import AsyncSelect from 'react-select/lib/Async'; Otherwise, AsyncSelect works the same as Select, as follows: <AsyncSelect value={value} onChange={value => setValue(value)} textFieldProps={{ label: 'Team', InputLabelProps: { shrink: true } }} {...{ ...props, classes }} /> How does it work? The only difference between a Select autocomplete and an AsyncSelect autocomplete is what happens while the request to the API is pending. Here is what the autocomplete looks like while this is happening: As the user types the CircularProgress component is rendered to the right, while the loading message is rendered in the menu using a Typography component. Highlighting search results When the user starts typing in an autocomplete and the results are displayed in the dropdown, it isn't always obvious how a given item matches the search criteria. You can help your users better understand the results by highlighting the matched portion of the string value. How to do it? You'll want to use two functions from the autosuggest-highlight package to help highlight the text presented in the autocomplete dropdown, as follows: import match from 'autosuggest-highlight/match'; import parse from 'autosuggest-highlight/parse'; Now, you can build a new component that will render the item text, highlighting as and when necessary, as follows: const ValueLabel = ({ label, search }) => { const matches = match(label, search); const parts = parse(label, matches); return parts.map((part, index) => part.highlight ? ( <span key={index} style={{ fontWeight: 500 }}> {part.text} </span> ) : ( <span key={index}>{part.text}</span> ) ); }; The end result is that ValueLabel renders an array of span elements, determined by the parse() and match() functions. One of the spans will be bolded if part.highlight is true. Now, you can use ValueLabel in the Option component, as follows: const Option = props => ( <MenuItem buttonRef={props.innerRef} selected={props.isFocused} component="div" style={{ fontWeight: props.isSelected ? 500 : 400 }} {...props.innerProps} > <ValueLabel label={props.children} search={props.selectProps.inputValue} /> </MenuItem> ); How does it work? Now, when you search for values in the autocomplete text input, the results will highlight the search criteria in each item, as follows: This article helped you implement autocompletion in your Material UI React application.  Then we implemented multi-value selection and saw how to better serve the autocomplete data through an API endpoint. If you found this post useful, do check out the book, React Material-UI Cookbook by Adam Boduch.  This book will help you build modern-day applications by implementing Material Design principles in React applications using Material-UI. How to create a native mobile app with React Native [Tutorial] Reactive programming in Swift with RxSwift and RxCocoa [Tutorial] How to build a Relay React App [Tutorial]
Read more
  • 0
  • 0
  • 31680
Banner background image

article-image-doteveryone-report-claims-the-absence-of-ethical-frameworks-and-support-mechanisms-could-lead-to-a-brain-drain-in-the-u-k-tech-industry
Richard Gall
13 May 2019
5 min read
Save for later

Doteveryone report claims the absence of ethical frameworks and support mechanisms could lead to a 'brain drain' in the U.K. tech industry

Richard Gall
13 May 2019
5 min read
Ethics in the tech industry is an issue that's drawing a lot of attention from policy makers and business leaders alike. But the opinions of the people who actually work in tech have, for a long time, been absent from the conversation. However, UK-based think tank Doteveryone are working hard to change that. In a report published today - People Power and Technology: The Tech Workers' View - the think tank sheds light on tech workers' perspectives on the impact of their work on society. It suggests that not only that there are significant tensions between the people who do the work, and the people they're working for, but also that much of the tech workforce would welcome more regulation and support within the tech industry. Backed by responses from more than 1,000 tech workers, the Doteveryone report unearthed some interesting trends: 18% of respondents has left their job because they felt uncomfortable about the ethics of certain ethical decisions. 45% of respondents said they thought there was too little regulation of tech (14% said there was too much). Those working in artificial intelligence had more experience or a more acute awareness of ethical issues - 27% of respondents said they had left their job as a consequence of a decision (compared with 18% overall). What are the biggest ethical challenges for tech workers, according to Doteveryone? Respondents to the survey were concerned about a wide range of issues. Many of these are ones that have been well-documented over the last year or so, but the report nevertheless demonstrates how those closest to the technology that's driving significant  - and potentially negative - change see those issues. One respondent quoted in the survey mentioned the issue of testing - something that often gets missed in broad and abstract conversations about privacy and automation. They said that their product "was not properly tested, and the company was too quick" to put it on the market. Elsewhere, another respondent discussed a problem with the metrics being chased by software companies, where clicks and usage are paramount. "Some features tend to be addictive," they said. Other respondents mentioned "the exploitation of personal data" and "mass unemployment," indicating that even those at the center of the tech industry harbour serious concerns about the impact of technology on society. However, one of the most interesting was one that we have discussed a lot - the importance of communication. "Too much technology less communication [sic] between people," said one respondent. In a nutshell, that seems to encapsulate Doteveryone's overall mission - a greater focus on the human element of technology, and resisting the seductions of technological solutionism. Read next: Mozilla and Google Chrome refuse to support Gab’s Dissenter extension for violating acceptable use policy How the absence of an ethical framework could cause brain drain in the U.K. tech sector The Doteveryone report points out that the challenges tech workers face, and the level of discomfort they feel in their respective jobs could lead to retention problems across the industry. The report states: The UK tech industry has major concerns about the availability of staff. 93% of employers have struggled to recruit to tech roles in the past year, with shortages most acute for management and experienced professionals. Brexit is expected to exacerbate these issues. Each lost tech worker is estimated to cost a company over £30,000. Our findings show that potentially irresponsible technology practices are a significant factor for retention and it’s vital that these are addressed for the industry to thrive. The key takeaway, then, is that the instability and pressure that comes with working in a job or industry where innovation is everything requires a frameworks and support mechanisms. Without them, the people actually doing the work in tech will not only find it harder to work responsibly, it will also take its toll on them personally. With issues around burnout in the tech industry stealing the headlines in recent months (think about the protests around China's 996 culture) this confirms the importance of working together to develop new ways of supporting and protecting each other. Read next: Artist Holly Herndon releases an album featuring an artificial intelligence ‘musician’ How tech workers are currently negotiating ethics The report finds that the when negotiating ethical issues, tech workers primarily turn to their "own moral compass." It shows that formal support structures feature very low, with only 29% turning to company policy, and 27% to industry standards. But this doesn't mean tech workers are only guided by their conscience - many use the internet (38%) or discuss with colleagues (35%). [caption id="attachment_27757" align="alignright" width="799"] via Doteveryone.org[/caption]                   What does Doteveryone argue needs to be done? To combat these issues, Doteveryone is very clear in its recommendations. It argues that businesses need much more transparency in their processes for raising ethical issues, increased investment in training and resources so tech workers are better equipped to engage with ethical decision making in their work, and to work collaboratively to build industry-wide standards. But the report doesn't only call on businesses to act - it also calls for government to "provide incentives for responsible innovation." This is particularly interesting, as it is a little different to the continuing mantra of regulation. It arguably changes the discussion, focusing less on rules and restrictions, and much more on actively encouraging a human-centered approach to technology and business. How the conversation around ethics and technology evolves in the years to come remains to be seen. But whatever happens, it's certainly essential that the views of those at the center of the industry are heard, and that they are empowered, as the report says, "to do their best work."
Read more
  • 0
  • 0
  • 3406

article-image-does-it-make-sense-to-talk-about-devops-engineers-or-devops-tools
Richard Gall
10 May 2019
6 min read
Save for later

Does it make sense to talk about DevOps engineers or DevOps tools?

Richard Gall
10 May 2019
6 min read
DevOps engineers are in high demand - the job represents an engineering unicorn, someone that understands both development and operations and can help to foster a culture where the relationship between the two is almost frictionless. But there's some debate as to whether it makes sense to talk about a DevOps engineer at all. If DevOps is a culture of a set of practices that improves agility and empowers engineers to take more ownership over their work, should we really be thinking about DevOps as a single job that someone can simply train for? The quotes in this piece are taken from DevOps Paradox by Viktor Farcic, which will be published in June 2019. The book features interviews with a diverse range of figures drawn from across the DevOps world. Is DevOps engineer a 'real' job or just recruitment spin? Nirmal Mehta (@normalfaults), Technology Consultant at Booz Allen Hamilton, says "There's no such thing as a DevOps engineer. There shouldn't even be a DevOps team, because to me DevOps is more of a cultural and philosophical methodology, a process, and a way of thinking about things and communicating within an IT organization..." Mehta is cyncical about organizations that put out job descriptions asking for DevOps engineers. It is, he argues, a way of cutting costs - a way of simply doing more with less. "A DevOps engineer is just a job posting that signals an organization wants to hire one less person to do twice as much work rather than hire both a developer and an operator." This view is echoed by other figures associated with the DevOps world. Mike Kail (@mdkail), CTO at Everest, says "I certainly don't view DevOps as a tool or a job title. In my view, at the core, it's a cultural approach to leveraging automation and orchestration to streamline both code development, infrastructure, application deployments and subsequently, the managing of those resources." Similarly, Damian Duportal (@DamienDuportal), Træfik's Developer Advocate, says "there is no such thing as a DevOps engineer or even a DevOps team.  The main purpose of DevOps is to focus on value, finding the optimal for the organization, and the value it will bring." For both Duportal and Kail, then, DevOps is primarily a cultural thing, something which needs to be embedded inside the practices of an organization. Is it useful to talk about a DevOps team? There are big question marks over the concept of a DevOps engineer. But what about a specific team? It's all well and good talking about organizational philosophy, but how do you actually affect change in a practical manner? Julian Simpson (@builddoctor), Neo4J's Global IT Manager is sceptical about the concept of a DevOps team: “Can we have something called a DevOps team? I don't believe so. You might spin up a team to solve a DevOps problem, but then I wouldn't even say we specifically have a DevOps problem. I'd say you just have a problem." DevOps consultant Chris Riley (@HoardingInfo) has a similar take, saying: “DevOps Engineer as a title makes sense to me, but I don't think you necessarily have DevOps departments, nor do you seek that out. Instead, I think DevOps is a principle that you spread throughout your entire development organization. Rather, you look to reform your organization in a way that supports those initiatives versus just saying that we need to build this DevOps unit, and there we go, we're done, we're DevOps. Because by doing that you really have to empower that unit and most organizations aren't willing to do that." However, Red Hat Solutions Architect Wian Vos  (@wianvos) has a different take. For Vos the idea of a DevOps team is actually crucial if you are to cultivate a DevOps mindset inside your organization: "Imagine... you and I were going to start a company. We're going to need a DevOps team because we have a burning desire to put out this awesome application. The questions we have when we're putting together a DevOps team is both ‘Who are we hiring?’ and ‘What are we hiring for? Are we going to hire DevOps engineers? No. In that team, we want the best application developers, the best tester, and maybe we want a great infrastructure guy and a frontend/backend developer. I want people with specific roles who fit together as a team to be that DevOps team." For Vos, it's not so much about finding and hiring DevOps engineers - people with a specific set of skills and experience - but rather building a team that's constructed in such a way that it can put DevOps principles into practice. Is there such a thing as a DevOps tool? One of the interesting things about DevOps is that the debate seems to lead you into a bit of a bind. It's almost as if the more concrete we try and make it - turning it into a job, or a team - the less useful it becomes. This is particularly true when we consider tooling. Surely thinking about DevOps technologically, rather than speculatively makes it more real? In general, it appears there is a consensus against the idea of DevOps tools. On this point Julian Simpson said "my original thinking about the movement from 2009 onwards, when the name was coined, was that it would be about collaboration and perhaps the tools would sort of come out of that collaboration." James Turnbull (@kartar), CEO of Rethink Robotics is critical of the notion of DevOps tools. He says "I don't think there are such things as DevOps tools. I believe there are tools that make the process of being a cross-functional team better... Any tool that facilitates building that cross-functionality is probably a DevOps tool to the point where the term is likely meaningless." When it comes to DevOps, everyone's still learning With even industry figures disagreeing on what terms mean, or which ones are relevant, it's pretty clear that DevOps will remain a field that's contested and debated. But perhaps this is important - if we expect it to simply be a solution to the engineering challenges we face, it's already failed as a concept. However, if we understand it as a framework or mindset for solving problems then that is when it acquires greater potency. Viktor Farcic is a Developer Advocate at CloudBees, a member of the Google Developer Experts and Docker Captains groups, and published author. His big passions are DevOps, Microservices, Continuous Integration, Delivery and Deployment (CI/CD) and Test-Driven Development (TDD).
Read more
  • 0
  • 0
  • 4631

article-image-iclr-2019-highlights-algorithmic-fairness-ai-for-social-good-climate-change-protein-structures-gan-magic-adversarial-ml-and-much-more
Amrata Joshi
09 May 2019
7 min read
Save for later

ICLR 2019 Highlights: Algorithmic fairness, AI for social good, climate change, protein structures, GAN magic, adversarial ML and much more

Amrata Joshi
09 May 2019
7 min read
The ongoing ICLR 2019 (International Conference on Learning Representations) has brought a pack full of surprises and key specimens of innovation. The conference started on Monday, this week and it’s already the last day today! This article covers the highlights of ICLR 2019 and introduces you to the ongoing research carried out by experts in the field of deep learning, data science, computational biology, machine vision, speech recognition, text understanding, robotics and much more. The team behind ICLR 2019, invited papers based on Unsupervised objectives for agents, Curiosity and intrinsic motivation, Few shot reinforcement learning, Model-based planning and exploration, Representation learning for planning, Learning unsupervised goal spaces, Unsupervised skill discovery and Evaluation of unsupervised agents. https://twitter.com/alfcnz/status/1125399067490684928 ICLR 2019, sponsored by Google marks the presence of 200 researchers contributing to and learning from the academic research community by presenting papers and posters. ICLR 2019 Day 1 highlights: Neural network, Algorithmic fairness, AI for social good and much more Algorithmic fairness https://twitter.com/HanieSedghi/status/1125401294880083968 The first day of the conference started with a talk on Highlights of Recent Developments in Algorithmic Fairness by Cynthia Dwork, an American computer scientist at Harvard University. She focused on "group fairness" notions that address the relative treatment of different demographic groups. And she talked on research in the ML community that explores fairness via representations. The investigation of scoring, classifying, ranking, and auditing fairness was also discussed in this talk by Dwork. Generating high fidelity images with Subscale Pixel Networks and Multidimensional Upscaling https://twitter.com/NalKalchbrenner/status/1125455415553208321 Jacob Menick, a senior research engineer at Google, Deep Mind and Nal Kalchbrenner, staff research scientist and co-creator of the Google Brain Amsterdam research lab talked on Generating high fidelity images with Subscale Pixel Networks and Multidimensional Upscaling. They talked about the challenges involved in generating the images and how they address this issue with the help of Subscale Pixel Network (SPN). It is a conditional decoder architecture that helps in generating an image as a sequence of image slices of equal size. They also explained how Multidimensional Upscaling is used to grow an image in both size and depth via intermediate stages corresponding to distinct SPNs. There were in all 10 workshops conducted on the same day based on AI and deep learning covering topics such as, The 2nd Learning from Limited Labeled Data (LLD) Workshop: Representation Learning for Weak Supervision and Beyond Deep Reinforcement Learning Meets Structured Prediction AI for Social Good Debugging Machine Learning Models The first day also witnessed a few interesting talks on neural networks covering topics such as The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, How Powerful are Graph Neural Networks? etc. Overall the first day was quite enriching and informative. ICLR 2019 Day 2 highlights: AI in climate change, Protein structure, adversarial machine learning, CNN models and much more AI’s role in climate change https://twitter.com/natanielruizg/status/1125763990158807040 Tuesday, also the second day of the conference, started with an interesting talk on Can Machine Learning Help to Conduct a Planetary Healthcheck? by Emily Shuckburgh, a Climate scientist and deputy head of the Polar Oceans team at the British Antarctic Survey. She talked about the sophisticated numerical models of the Earth’s systems which have been developed so far based on physics, chemistry and biology. She then highlighted a set of "grand challenge" problems and discussed various ways in which Machine Learning is helping to advance our capacity to address these. Protein structure with a differentiable simulator On the second day of ICLR 2019, Chris Sander, computational biologist, John Ingraham, Adam J Riesselman, and Debora Marks from Harvard University, talked on Learning protein structure with a differentiable simulator. They about the protein folding problem and their aim to bridge the gap between the expressive capacity of energy functions and the practical capabilities of their simulators by using an unrolled Monte Carlo simulation as a model for data. They also composed a neural energy function with a novel and efficient simulator which is based on Langevin dynamics for building an end-to-end-differentiable model of atomic protein structure given amino acid sequence information. They also discussed certain techniques for stabilizing backpropagation and demonstrated the model's capacity to make multimodal predictions. Adversarial Machine Learning https://twitter.com/natanielruizg/status/1125859734744117249 Day 2 was long and had Ian Goodfellow, a machine learning researcher and inventor of GANs, to talk on Adversarial Machine Learning. He talked about supervised learning works and making machine learning private, getting machine learning to work for new tasks and also reducing the dependency on large amounts of labeled data. He then discussed how the adversarial techniques in machine learning are involved in the latest research frontiers. Day 2 covered poster presentation and a few talks on Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset,  Learning to Remember More with Less Memorization, Learning to Remember More with Less Memorization, etc. ICLR 2019 Day 3 highlights: GAN, Autonomous learning and much more Developmental autonomous learning: AI, Cognitive Sciences and Educational Technology https://twitter.com/drew_jaegle/status/1125522499150721025 Day 3 of ICLR 2019 started with Pierre-Yves Oudeyer’s, research director at Inria talk on Developmental Autonomous Learning: AI, Cognitive Sciences and Educational Technology. He presented a research program that focuses on computational modeling of child development and learning mechanisms. He then discussed the several developmental forces that guide exploration in large real-world spaces. He also talked about the models of curiosity-driven autonomous learning that enables machines to sample and explore their own goals and learning strategies. He then explained how these models and techniques can be successfully applied in the domain of educational technologies. Generating knockoffs for feature selection using Generative Adversarial Networks (GAN) Another interesting topic on the third day of ICLR 2019 was Generating knockoffs for feature selection using Generative Adversarial Networks (GAN) by James Jordon from Oxford University, Jinsung Yoon from California University, and Mihaela Schaar Professor at UCLA. The experts talked about the Generative Adversarial Networks framework that helps in generating knockoffs with no assumptions on the feature distribution. They also talked about the model they created which consists of 4 networks, a generator, a discriminator, a stability network and a power network. They further demonstrated the capability of their model to perform feature selection. Followed by few more interesting topics like Deterministic Variational Inference for Robust Bayesian Neural Networks, there were series of poster presentations. ICLR 2019 Day 4 highlights: Neural networks, RNN, neuro-symbolic concepts and much more Learning natural language interfaces with neural models Today’s focus was more on neural models and neuro symbolic concepts. The day started with a talk on Learning natural language interfaces with neural models by Mirella Lapata, a computer scientist. She gave an overview of recent progress on learning natural language interfaces which allow users to interact with various devices and services using everyday language. She also addressed the structured prediction problem of mapping natural language utterances onto machine-interpretable representations. She further outlined the various challenges it poses and described a general modeling framework based on neural networks which tackle these challenges. Ordered neurons: Integrating tree structures into Recurrent Neural Networks https://twitter.com/mmjb86/status/1126272417444311041 The next interesting talk was on Ordered neurons: Integrating tree structures into Recurrent Neural Networks by Professors Yikang Shen, Aaron Courville and Shawn Tan from Montreal University, and, Alessandro Sordoni, a researcher at Microsoft. In this talk, the experts focused on how they proposed a new RNN unit: ON-LSTM, which achieves good performance on four different tasks including language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference. The last day of ICLR 2019 was exciting and helped the researchers present their innovations and attendees got a chance to interact with the experts. To have a complete overview of each of these sessions, you can head over to ICLR’s Facebook page. Paper in Two minutes: A novel method for resource efficient image classification Google I/O 2019 D1 highlights: smarter display, search feature with AR capabilities, Android Q, linguistically advanced Google lens and more Google I/O 2019: Flutter UI framework now extended for Web, Embedded, and Desktop
Read more
  • 0
  • 0
  • 4454
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-deepmind-alphago-zero-game-changer-for-ai-research
Guest Contributor
09 May 2019
10 min read
Save for later

Why DeepMind AlphaGo Zero is a game changer for AI research

Guest Contributor
09 May 2019
10 min read
DeepMind, a London based artificial intelligence (AI) company currently owned by Alphabet, recently made great strides in AI with its AlphaGo program. It all began in October 2015 when the program beat the European Go champion Fan Hui 5-0, in a game of Go. This was the very first time an AI defeated a professional Go player. Earlier, computers were only known to have played Go at the "amateur" level. Then, the company made headlines again in 2016 after its AlphaGo program beat Lee Sedol, a professional Go player (a world champion) with a score of 4-1 in a five-game match. Furthermore, in late 2017, an improved version of the program called AlphaGo Zero defeated AlphaGo 100 games to 0. The best part? AlphaGo Zero's strategies were self-taught i.e it was trained without any data from human games. AlphaGo Zero was able to defeat its predecessor in only three days time with lesser processing power than AlphaGo. However, the original AlphaGo, on the other hand required months to learn how to play. All these facts beg the questions: what makes AlphaGo Zero so exceptional? Why is it such a big deal? How does it even work? So, without further ado, let’s dive into the what, why, and how of DeepMind’s AlphaGo Zero. What is DeepMind AlphaGo Zero? Simply put, AlphaGo Zero is the strongest Go program in the world (with the exception of AlphaZero). As mentioned before, it monumentally outperforms all previous versions of AlphaGo. Just check out the graph below which compares the Elo rating of the different versions of AlphaGo. Source: DeepMind The Elo rating system is a method for calculating the relative skill levels of players in zero-sum games such as chess and Go. It is named after its creator Arpad Elo, a Hungarian-American physics professor. Now, all previous versions of AlphaGo were trained using human data. The previous versions learned and improved upon the moves played by human experts/professional Go players. But AlphaGo Zero didn’t use any human data whatsoever. Instead, it had to learn completely from playing against itself. According to DeepMind's Professor David Silver, the reason that playing against itself enables it to do so much better than using strong human data is that AlphaGo always has an opponent of just the right level. So it starts off extremely naive, with perfectly random play. And yet at every step of the learning process, it has an opponent (a “sparring partner”) that’s exactly calibrated to its current level of performance. That is, to begin with, these players are terribly weak but over time they become progressively stronger and stronger. Why is reinforcement learning such a big deal? People tend to assume that machine learning is all about big data and massive amounts of computation. But actually, with AlphaGo Zero, AI scientists at DeepMind realized that algorithms matter much more than the computing processing power or data availability. AlphaGo Zero required less computation than previous versions and yet it was able to perform at a much higher level due to using much more principled algorithms than before. It is a system which is trained completely from scratch, starting from random behavior, and progressing from first principles to really discover tabula rasa, in playing the game of Go. It is, therefore, no longer constrained by the limits of human knowledge. Note that AlphaGo Zero did not use zero-shot learning which essentially is the ability of the machine to solve a task despite not having received any training for that task. How does it work? AlphaGo Zero is able to achieve all this by employing a novel form of reinforcement learning, in which AlphaGo Zero becomes its own teacher. As explained previously, the system starts off with a single neural network that knows absolutely nothing about the game of Go. By combining this neural network with a powerful search algorithm, it then plays games against itself. As it plays more and more games, the neural network is updated and tuned to predict moves, and even the eventual winner of the games. This revised neural network is then recombined with the search algorithm to generate a new, stronger version of AlphaGo Zero, and the process repeats. With each iteration, the performance of the system enhances with each iteration, and the quality of the self-play games’ advances, leading to increasingly accurate neural networks and ever-more powerful versions of AlphaGo Zero. Now, let’s dive into some of the technical details that make this version of AlphaGo so much better than all its forerunners. AlphaGo Zero's neural network was trained using TensorFlow, with 64 GPU workers and 19 CPU parameter servers. Only four Tensor Processing Units (TPUs) were used for inference. And of course, the neural network initially knew nothing about Go beyond the rules. Both AlphaGo and AlphaGo Zero took a general approach to play Go. Both evaluated the Go board and chose moves using a combination of two methods: Conducting a “lookahead” search: This means looking ahead several moves by simulating games, and hence seeing which current move is most likely to lead to a “good” position in the future. Assessing positions based on an “intuition” of whether a position is “good” or “bad”  and is likely to result in a win or a loss. Go is a truly intricate game which means computers can’t merely search all possible moves using a brute force approach to discover the best one. Method 1: Lookahead Before AlphaGo, all the finest Go programs tackled this issue by using “Monte Carlo Tree Search” or MCTS. This process involves initially exploring numerous possible moves on the board and then focusing this search over time as certain moves are found to be more likely to result in wins than others. Source: LOC Both AlphaGo and AlphaGo Zero apply a fairly elementary version of MCTS for their “lookahead” to correctly maintain the tradeoff between exploring new sequences of moves or more deeply explore already-explored sequences. Although MCTS has been at the heart of all effective Go programs preceding AlphaGo, it was DeepMind’s smart coalescence of this method with a neural network-based “intuition” that enabled it to attain superhuman performance. Method 2: Intuition DeepMind’s pivotal innovation with AlphaGo was to utilize deep neural networks to identify the state of the game and then use this knowledge to effectively guide the search of the MCTS. In particular, they trained networks that could record: The current board position Which player was playing The sequence of recent moves (in order to rule out certain moves as “illegal”) With this data, the neural networks could propose: Which move should be played If the current player is likely to win or not So how did DeepMind train neural networks to do this? Well, AlphaGo and AlphaGo Zero used rather different approaches in this case. AlphaGo had two separately trained neural networks: Policy Network and Value Network. Source: AlphaGo’s Nature Paper DeepMind then fused these two neural networks with MCTS  —  that is, the program’s “intuition” with its brute force “lookahead” search — in an ingenious way. It used the network that had been trained to predict: Moves to guide which branches of the game tree to search Whether a position was “winning” to assess the positions it encountered during its search This let AlphaGo to intelligently search imminent moves and eventually beat the world champion Lee Sedol. AlphaGo Zero, however, took this principle to the next level. Its neural network’s “intuition” was trained entirely differently from that of AlphaGo. More specifically: The neural network was trained to play moves that exhibited the improved evaluations from performing the “lookahead” search The neural network was tweaked so that it was more likely to play moves like those that led to wins and less likely to play moves similar to those that led to losses during the self-play games Much was made of the fact that no games between humans were used to train AlphaGo Zero. Thus, for a given state of a Go agent, it can constantly be made smarter by performing MCTS-based lookahead and using the results of that lookahead to upgrade the agent. This is how AlphaGo Zero was able to perpetually improve, from when it was an “amateur” all the way up to when it better than the best human players. Moreover, AlphaGo Zero’s neural network architecture can be referred to as a “two-headed” architecture. Source: Hacker Noon Its first 20 layers were “blocks” of a typically seen in modern neural net architectures. These layers were followed by two “heads”: One head that took the output of the first 20 layers and presented probabilities of the Go agent making certain moves Another head that took the output of the first 20 layers and generated a probability of the current player winning. What’s more, AlphaGo Zero used a more “state of the art” neural network architecture as opposed to AlphaGo. Particularly, it used a “residual” neural network architecture rather than a plainly “convolutional” architecture. Deep residual learning was pioneered by Microsoft Research in late 2015, right around the time work on the first version of AlphaGo would have been concluded. So, it is quite reasonable that DeepMind did not use them in the initial AlphaGo program. Notably, each of these two neural network-related acts —  switching from separate-convolutional to the more advanced dual-residual architecture and using the “two-headed” neural network architecture instead of separate neural networks  —  would have resulted in nearly half of the increase in playing strength as was realized when both were coupled. Source: AlphaGo’s Nature Paper Wrapping it up According to DeepMind: “After just three days of self-play training, AlphaGo Zero emphatically defeated the previously published version of AlphaGo - which had itself defeated 18-time world champion Lee Sedol - by 100 games to 0. After 40 days of self-training, AlphaGo Zero became even stronger, outperforming the version of AlphaGo known as “Master”, which has defeated the world's best players and world number one Ke Jie. Over the course of millions of AlphaGo vs AlphaGo games, the system progressively learned the game of Go from scratch, accumulating thousands of years of human knowledge during a period of just a few days. AlphaGo Zero also discovered new knowledge, developing unconventional strategies and creative new moves that echoed and surpassed the novel techniques it played in the games against Lee Sedol and Ke Jie.” Further, the founder and CEO of DeepMind, Dr. Demis Hassabis believes AlphaGo's algorithms are likely to most benefit to areas that need an intelligent search through an immense space of possibilities. Author Bio Gaurav is a Senior SEO and Content Marketing Analyst at The 20 Media, a Content Marketing agency that specializes in data-driven SEO. He has more than seven years of experience in Digital Marketing and along with that loves to read and write about AI, Machine Learning, Data Science and much more about the emerging technologies. In his spare time, he enjoys watching movies and listening to music. Connect with him on Twitter and LinkedIn. DeepMind researchers provide theoretical analysis on recommender system, ‘echo chamber’ and ‘filter bubble effect’ What if AIs could collaborate using human-like values? DeepMind researchers propose a Hanabi platform. Google DeepMind’s AI AlphaStar beats StarCraft II pros TLO and MaNa; wins 10-1 against the gamers  
Read more
  • 0
  • 0
  • 12645

article-image-which-python-framework-is-best-for-building-restful-apis-django-or-flask
Vincy Davis
07 May 2019
9 min read
Save for later

Which Python framework is best for building RESTful APIs? Django or Flask?

Vincy Davis
07 May 2019
9 min read
Python is one of the top-rated programming languages. It's also known for its less-complex syntax, and its high-level, object-oriented, robust, and general-purpose programming. Python is the top choice for any first-time programmer. Since its release in 1991, Python has evolved and powered by several frameworks for web application development, scientific and mathematical computing, and graphical user interfaces to the latest REST API frameworks. This article is an excerpt taken from the book, 'Hands-On RESTful API Design Patterns and Best Practices' written by Harihara Subramanian and Pethura Raj. This book covers design strategy, essential and advanced Restful API Patterns, Legacy Modernization to Microservices centric apps. In this article, we'll explore two comprehensive frameworks, Django and Flask, so that you can choose the best one for developing your RESTful API. Django Django is a web framework also available as open source with the BSD license, designed to help developers create their web app very quickly as it takes care of additional web-development needs. It includes several packages (also known as applications) to handle typical web-development tasks, such as authentication, content administration, scaffolding, templates, caching, and syndication. Let's use the Django REST Framework (DRF) built with Python, and use it for REST API development and deployment. Django Rest Framework DRF is an open source, well-matured Python and Django library intended to help APP developers build sophisticated web APIs. DRF's modular, flexible, and customizable architecture makes the development of both simple, turnkey API endpoints and complicated REST constructs possible. The goal of DRF is to divide a model, generalize the wire representation, such as JSON or XML, and customize a set of class-based views to satisfy the specific API endpoint using a serializer that describes the mapping between views and API endpoints. Core features Django has many distinct features including: Web-browsable API This feature enhances the REST API developed with DRF. It has a rich interface, and the web-browsable API supports multiple media types too. The browsable API does mean that the APIs we build will be self-describing and the API endpoints that we create as part of the REST services and return JSON or HTML representations. The interesting fact about the web-browsable API is that we can interact with it fully through the browser, and any endpoint that we interact with using a programmatic client will also be capable of responding with a browser-friendly view onto the web-browsable API. Authentication One of the main attractive features of Django is authentication; it supports broad categories of authentication schemes, from basic authentication, token authentication, session authentication, remote user authentication, to OAuth Authentication. It also supports custom authentication schemes if we wish to implement one. DRF runs the authentication scheme at the start of the view, that is, before any other code is allowed to proceed. DRF determines the privileges of the incoming request from the permission and throttling policies and then decides whether the incoming request can be allowed or disallowed with the matched credentials. Serialization and deserialization Serialization is the process of converting complex data, such as querysets and model instances, into native Python datatypes. Converting facilitates the rendering of native data types, such as JSON or XML. DRF supports serialization through serializers classes. The serializers of DRF are similar to Django's Form and ModelForm classes. It provides a serializer class, which helps to control the output of responses. The DRF ModelSerializer classes provide a simple mechanism with which we can create serializers that deal with model instances and querysets. Serializers also do deserialization, that is, serializers allow parsed data that needs to be converted back into complex types. Also, deserialization happens only after validating the incoming data. Other noteworthy features Here are some other noteworthy features of the DRF: Routers: The DRF supports automatic URL routing to Django and provides a consistent and straightforward way to wire the view logic to a set of URLs Class-based views: A dominant pattern that enables the reusability of common functionalities Hyperlinking APIs: The DRF supports various styles (using primary keys, hyperlinking between entities, and so on) to represent the relationship between entities Generic views: Allows us to build API views that map to the database models DRF has many other features such as caching, throttling, testing, etc. Benefits of the DRF Here are some of the benefits of the DRF: Web-browsable API Authentication policies Powerful serialization Extensive documentation and excellent community support Simple yet powerful Test coverage of source code Secure and scalable Customizable Drawbacks of the DRF Here are some facts that may disappoint some Python app developers who intend to use the DRF: Monolithic and components get deployed together Based on Django ORM Steep learning curve Slow response time Flask Flask is a microframework for Python developers based on Werkzeug (WSGI toolkit) and Jinja 2 (template engine). It comes under BSD licensing. Flask is very easy to set up and simple to use. Like other frameworks, it comes with several out-of-the-box capabilities, such as a built-in development server, debugger, unit test support, templating, secure cookies, and RESTful request dispatching. The powerful Flask  RESTful API framework is discussed below. Flask-RESTful Flask-RESTful is an extension for Flask that provides additional support for building REST APIs. You will never be disappointed with the time it takes to develop an API. Flask-Restful is a lightweight abstraction that works with the existing ORM/libraries. Flask-RESTful encourages best practices with minimal setup. Core features of Flask-RESTful Flask-RESTful comes with several built-in features. Django and Flask have many common RESTful frameworks, because they have almost the same supporting core features. The unique RESTful features of Flask is mentioned below. Resourceful routing The design goal of Flask-RESTful is to provide resources built on top of Flask pluggable views. The pluggable views provide a simple way to access the HTTP methods. Consider the following example code: class Todo(Resource): def get(self, user_id): .... def delete(self, user_id): .... def put(self, user_id): args = parser.parse_args() .... Restful request parsing Request parsing refers to an interface, modeled after the Python parser interface for command-line arguments, called argparser. The RESTful request parser is designed to provide uniform and straightforward access to any variable that comes within the (flask.request) request object. Output fields In most cases, app developers prefer to control rendering response data, and Flask-RESTful provides a mechanism where you can use ORM models or even custom classes as an object to render. Another interesting fact about this framework is that app developers don't need to worry about exposing any internal data structures as its let one format and filter the response objects. So, when we look at the code, it'll be evident which data would go for rendering and how it'll be formatted. Other noteworthy features Here are some other noteworthy features of Flask-RESTful: API: This is the main entry point for the restful API, which we'll initialize with the Flask application. ReqParse: This enables us to add and parse multiple arguments in the context of the single request. Input: A useful functionality, it parses the input string and returns true or false depending on the Input. If the input is from the JSON body,  the type is already native Boolean and passed through without further parsing. Benefits of the Flask framework Here are some of the benefits of Flask framework: Built-in development server and debugger Out-of-the-box RESTful request dispatching Support for secure cookies Integrated unit-test support Lightweight Very minimal setup Faster (performance) Easy NoSQL integration Extensive documentation Drawbacks of Flask Here are some of Flask and Flask-RESTful's disadvantages: Version management (managed by developers) No brownie points as it doesn't have browsable APIs May incur a steep learning curve Frameworks – a table of reference The following table provides a quick reference of a few other prominent micro-frameworks, their features, and supported programming languages: Language Framework Short description Prominent features Java Blade Fast and elegant MVC framework for Java8 Lightweight High performance Based on the MVC pattern RESTful-style router interface Built-in security Java/Scala Play Framework High-velocity Reactive web framework for Java and Scala Lightweight, stateless, and web-friendly architecture Built on Akka Supports predictable and minimal resource-consumption for highly-scalable applications Developer-friendly Java Ninja Web Framework Full-stack web framework Fast Developer-friendly Rapid prototyping Plain vanilla Java, dependency injection, first-class IDE integration Simple and fast to test (mocked tests/integration tests) Excellent build and CI support Clean codebase – easy to extend Java RESTEASY JBoss-based implementation that integrates several frameworks to help to build RESTful Web and Java applications Fast and reliable Large community Enterprise-ready Security support Java RESTLET A lightweight and comprehensive framework based on Java, suitable for both server and client applications. Lightweight Large community Native REST support Connectors set JavaScript Express.js Minimal and flexible Node.js-based JavaScript framework for mobile and web applications HTTP utility methods Security updates Templating engine PHP Laravel An open source web-app builder based on PHP and the MVC architecture pattern Intuitive interface Blade template engine Eloquent ORM as default Elixir Phoenix (Elixir) Powered with the Elixir functional language, a reliable and faster micro-framework MVC-based High application performance Erlong virtual machine enables better use of resources Python Pyramid Python-based micro-framework Lightweight Function decorators Events and subscribers support Easy implementations and high productivity Summary It's evident that Python has two excellent frameworks. Depending on the choice of programming language you are intending to use and the required features, you can choose your type of framework to work on. If you are interested in learning more about the design strategy, guidelines and best practices of Restful API Patterns, you can refer to our book 'Hands-On RESTful API Design Patterns and Best Practices' here. Stack Overflow survey data further confirms Python’s popularity as it moves above Java in the most used programming language list. Svelte 3 releases with reactivity through language instead of an API Microsoft introduces Pyright, a static type checker for the Python language written in TypeScript
Read more
  • 0
  • 0
  • 37428

article-image-microsoft-build-2019-microsoft-showcases-new-updates-to-ms-365-platfrom-with-focus-on-ai-and-developer-productivity
Sugandha Lahoti
07 May 2019
10 min read
Save for later

Microsoft Build 2019: Microsoft showcases new updates to MS 365 platform with focus on AI and developer productivity

Sugandha Lahoti
07 May 2019
10 min read
At the ongoing Microsoft Build 2019 conference, Microsoft has announced a ton of new features and tool releases with a focus on innovation using AI and mixed reality with the intelligent cloud and the intelligent edge. In his opening keynote, Microsoft CEO Satya Nadella outlined the company’s vision and developer opportunity across Microsoft Azure, Microsoft Dynamics 365 and IoT Platform, Microsoft 365, and Microsoft Gaming. “As computing becomes embedded in every aspect of our lives, the choices developers make will define the world we live in,” said Satya Nadella, CEO, Microsoft. “Microsoft is committed to providing developers with trusted tools and platforms spanning every layer of the modern technology stack to build magical experiences that create new opportunity for everyone.” https://youtu.be/rIJRFHDr1QE Increasing developer productivity in Microsoft 365 platform Microsoft Graph data connect Microsoft Graphs are now powered with data connectivity, a service that combines analytics data from the Microsoft Graph with customers’ business data. Microsoft Graph data connect will provide Office 365 data and Microsoft Azure resources to users via a toolset. The migration pipelines are deployed and managed through Azure Data Factory. Microsoft Graph data connect can be used to create new apps shared within enterprises or externally in the Microsoft Azure Marketplace. It is generally available as a feature in Workplace Analytics and also as a standalone SKU for ISVs. More information here. Microsoft Search Microsoft Search works as a unified search experience across all Microsoft apps-  Office, Outlook, SharePoint, OneDrive, Bing and Windows. It applies AI technology from Bing and deep personalized insights surfaced by the Microsoft Graph to personalized searches. Other features included in Microsoft Search are: Search box displacement Zero query typing and key-phrase suggestion feature Query history feature, and personal search query history Administrator access to the history of popular searches for their organizations, but not to search history for individual users Files/people/site/bookmark suggestions Microsoft Search will begin publicly rolling out to all Microsoft 365 and Office 365 commercial subscriptions worldwide at the end of May. Read more on MS Search here. Fluid Framework As the name suggests Microsoft's newly launched Fluid framework allows seamless editing and collaboration between different applications. Essentially, it is a web-based platform and componentized document model that allows users to, for example, edit a document in an application like Word and then share a table from that document in Microsoft Teams (or even a third-party application) with real-time syncing. Microsoft says Fluid can translate text, fetch content, suggest edits, perform compliance checks, and more. The company will launch the software developer kit and the first experiences powered by the Fluid Framework later this year on Microsoft Word, Teams, and Outlook. Read more about Fluid framework here. Microsoft Edge new features Microsoft Build 2019 paved way for a bundle of new features to Microsoft’s flagship web browser, Microsoft Edge. New features include: Internet Explorer mode: This mode integrates Internet Explorer directly into the new Microsoft Edge via a new tab. This allows businesses to run legacy Internet Explorer-based apps in a modern browser. Privacy Tools: Additional privacy controls which allow customers to choose from 3 levels of privacy in Microsoft Edge—Unrestricted, Balanced, and Strict. These options limit third parties to track users across the web.  “Unrestricted” allows all third-party trackers to work on the browser. “Balanced” prevents third-party trackers from sites the user has not visited before. And “Strict” blocks all third-party trackers. Collections: Collections allows users to collect, organize, share and export content more efficiently and with Office integration. Microsoft is also migrating Edge as a whole over to Chromium. This will make Edge easier to develop for by third parties. For more details, visit Microsoft’s developer blog. New toolkit enhancements in Microsoft 365 Platform Windows Terminal Windows Terminal is Microsoft’s new application for Windows command-line users. Top features include: User interface with emoji-rich fonts and graphics-processing-unit-accelerated text rendering Multiple tab support and theming and customization features Powerful command-line user experience for users of PowerShell, Cmd, Windows Subsystem for Linux (WSL) and all forms of command-line application Windows Terminal will arrive in mid-June and will be delivered via the Microsoft Store in Windows 10. Read more here. React Native for Windows Microsoft announced a new open-source project for React Native developers at Microsoft Build 2019. Developers who prefer to use the React/web ecosystem to write user-experience components can now leverage those skills and components on Windows by using “React Native for Windows” implementation. React for Windows is under the MIT License and will allow developers to target any Windows 10 device, including PCs, tablets, Xbox, mixed reality devices and more. The project is being developed on GitHub and is available for developers to test. More mature releases will follow soon. Windows Subsystem for Linux 2 Microsoft rolled out a new architecture for Windows Subsystem for Linux: WSL 2 at the MSBuild 2019. Microsoft will also be shipping a fully open-source Linux kernel with Windows specially tuned for WSL 2. New features include massive file system performance increases (twice as much speed for file-system heavy operations, such as Node Package Manager install). WSL also supports running Linux Docker containers. The next generation of WSL arrives for Insiders in mid-June. More information here. New releases in multiple Developer Tools .NET 5 arrives in 2020 .NET 5 is the next major version of the .NET Platform which will be available in 2020. .NET 5 will have all .NET Core features as well as more additions: One Base Class Library containing APIs for building any type of application More choice on runtime experiences Java interoperability will be available on all platforms. Objective-C and Swift interoperability will be supported on multiple operating systems .NET 5 will provide both Just-in-Time (JIT) and Ahead-of-Time (AOT) compilation models to support multiple compute and device scenarios. .NET 5 also will offer one unified toolchain supported by new SDK project types as well as a flexible deployment model (side-by-side and self-contained EXEs) Detailed information here. ML.NET 1.0 ML.NET is Microsoft’s open-source and cross-platform framework that runs on Windows, Linux, and macOS and makes machine learning accessible for .NET developers. Its new version, ML.NET 1.0, was released at the Microsoft Build Conference 2019 yesterday. Some new features in this release are: Automated Machine Learning Preview: Transforms input data by selecting the best performing ML algorithm with the right settings. AutoML support in ML.NET is in preview and currently supports Regression and Classification ML tasks. ML.NET Model Builder Preview: Model Builder is a simple UI tool for developers which uses AutoML to build ML models. It also generates model training and model consumption code for the best performing model. ML.NET CLI Preview: ML.NET CLI is a dotnet tool which generates ML.NET Models using AutoML and ML.NET. The ML.NET CLI quickly iterates through a dataset for a specific ML Task and produces the best model. Visual Studio IntelliCode, Microsoft’s tool for AI-assisted coding Visual Studio IntelliCode, Microsoft’s AI-assisted coding is now generally available. It is essentially an enhanced IntelliSense, Microsoft’s extremely popular code completion tool. Intellicode is trained by using the code of thousands of open-source projects from GitHub that have at least 100 stars. It is available for C# and XAML for Visual Studio and Java, JavaScript, TypeScript, and Python for Visual Studio Code. IntelliCode also is included by default in Visual Studio 2019, starting in version 16.1 Preview 2. Additional capabilities, such as custom models, remain in public preview. Visual Studio 2019 version 16.1 Preview 2 Visual Studio 2019 version 16.1 Preview 2 release includes IntelliCode and the GitHub extensions by default. It also brings out of preview the Time Travel Debugging feature introduced with version 16.0. Also includes multiple performances and productivity improvements for .NET and C++ developers. Gaming and Mixed Reality Minecraft AR game for mobile devices At the end of Microsoft’s Build 2019 keynote yesterday, Microsoft teased a new Minecraft game in augmented reality, running on a phone. The teaser notes that more information will be coming on May 17th, the 10-year anniversary of Minecraft. https://www.youtube.com/watch?v=UiX0dVXiGa8 HoloLens 2 Development Edition and unreal engine support The HoloLens 2 Development Edition includes a HoloLens 2 device, $500 in Azure credits and three-months free trials of Unity Pro and Unity PiXYZ Plugin for CAD data, starting at $3,500 or as low as $99 per month. The HoloLens 2 Development Edition will be available for preorder soon and will ship later this year. Unreal Engine support for streaming and native platform integration will be available for HoloLens 2 by the end of May. Intelligent Edge and IoT Azure IoT Central new features Microsoft Build 2019 also featured new additions to Azure IoT Central, an IoT software-as-a-service solution. Better rules processing and customs rules with services like Azure Functions or Azure Stream Analytics Multiple dashboards and data visualization options for different types of users Inbound and outbound data connectors, so that operators can integrate with   systems Ability to add custom branding and operator resources to an IoT Central application with new white labeling options New Azure IoT Central features are available for customer trials. IoT Plug and Play IoT Plug and Play is a new, open modeling language to connect IoT devices to the cloud seamlessly without developers having to write a single line of embedded code. IoT Plug and Play also enable device manufacturers to build smarter IoT devices that just work with the cloud. Cloud developers will be able to find IoT Plug and Play enabled devices in Microsoft’s Azure IoT Device Catalog. The first device partners include Compal, Kyocera, and STMicroelectronics, among others. Azure Maps Mobility Service Azure Maps Mobility Service is a new API which provides real-time public transit information, including nearby stops, routes and trip intelligence. This API also will provide transit services to help with city planning, logistics, and transportation. Azure Maps Mobility Service will be in public preview in June. Read more about Azure Maps Mobility Service here. KEDA: Kubernetes-based event-driven autoscaling Microsoft and Red Hat collaborated to create KEDA, which is an open-sourced project that supports the deployment of serverless, event-driven containers on Kubernetes. It can be used in any Kubernetes environment — in any public/private cloud or on-premises such as Azure Kubernetes Service (AKS) and Red Hat OpenShift. KEDA has support for built-in triggers to respond to events happening in other services or components. This allows the container to consume events directly from the source, instead of routing through HTTP. KEDA also presents a new hosting option for Azure Functions that can be deployed as a container in Kubernetes clusters. Securing elections and political campaigns ElectionGuard SDK and Microsoft 365 for Campaigns ElectionGuard, is a free open-source software development kit (SDK) as an extension of Microsoft’s Defending Democracy Program to enable end-to-end verifiability and improved risk-limiting audit capabilities for elections in voting systems. Microsoft365 for Campaigns provides security capabilities of Microsoft 365 Business to political parties and individual candidates. More details here. Microsoft Build is in its 6th year and will continue till 8th May. The conference hosts over 6,000 attendees with early 500 student-age developers and over 2,600 customers and partners in attendance. Watch it live here! Microsoft introduces Remote Development extensions to make remote development easier on VS Code Docker announces a collaboration with Microsoft’s .NET at DockerCon 2019 How Visual Studio Code can help bridge the gap between full-stack development and DevOps [Sponsered by Microsoft]
Read more
  • 0
  • 0
  • 5792

article-image-pycon-2019-highlights-python-steering-council-discusses-the-changes-in-the-current-python-governance-structure
Bhagyashree R
07 May 2019
8 min read
Save for later

PyCon 2019 highlights: Python Steering Council discusses the changes in the current Python governance structure

Bhagyashree R
07 May 2019
8 min read
At the ongoing PyCon 2019 event, Python Steering Council shed some light on the recent changes in the Python governance structure and what these changes mean for the larger Python community. PyCon 2019 is the biggest gathering of developers and experts who work with the Python programming language. It is scheduled from May 1 to May 9 and is happening at Cleveland, Ohio. Backed by Python Software Foundation (PSF), this event hosts various tutorials, talks, summits, as well as a job fair. The Python Steering Council After a two week nomination period (January 7 to January 20), which was followed by a two week voting period (January 21 to February 4), five members were selected for the Python Steering Council: Guido van Rossum, the brilliant mind behind the Python programming language and the former Python BDFL (Benevolent Dictator for Life). Barry Warsaw, a Senior Staff Software Engineer at LinkedIn, and also the lead maintainer for Jython. Brett Cannon, a Principal Software Engineering Manager at Microsoft and a  Python core developer for over 15 years. Carol Willing, a Research Software Engineer for Project Jupyter, Python core developer, and PSF Fellow Nick Coghlan, a CPython Core developer for Python Software Foundation On why Guido van Rossum stepped down from being BDFL Since the dawn of the Python era, Guido van Rossum served as its Benevolent Dictator for Life (BDFL). It is a designation given to open-source software development leaders who have the final say in any kind of argument within the community. Guido stepped down from this designation last year in July and became a part of the Steering Council. Being a BDFL he was responsible for going through all the Python-ideas that might become controversial. Eventually, it ended up becoming his responsibility to take the final decision for PEPs which has already been discussed among the people with greater domain knowledge and expertise. After playing such a key authoritative role for nearly 30 years, Guido started experiencing, what is really common nowadays in the tech industry, the burnout syndrome. So, he finally took the right step of stepping down from his role as a BDFL and urging the core python core developers to discuss and decide amongst themselves the kind of governance structure they want for the community going forward. After months of intense research and debate, the team arrived at the decision of distributing these responsibilities among the five elected steering council members who have earned the trust of the Python community. He adds, “...that's pretty stressful and so I'm very glad that responsibility is now distributed over five experts who have more trust of the community because they've actually been voted in rather than just becoming the leader by happenstance.” Sharing his feelings about stepping down from the BDFL role, he said, “...when your kid goes off to college some of you may have experience with that I will soon have that experience. You're no longer directly involved in their lives maybe but you never stop worrying and that's how I feel about Python at the moment and that's why I nominated myself for the steering committee.” Changes in the PEP process with the new governance model The purpose behind Python Enhancement Proposals (PEPs) was to take away the burden from Guido of going through each and every email to understand what the proposal was about. He just needed to read one document listing all the pros and cons related to the proposal and then make a decision. This entire decision-making process was documented within the PEPs. With the growing Python community, this process became quite unattainable for Guido as all the decisions funneled through him. So, that is why the idea of BDFL delegate came up: an expert who will take care of the decision-making for a particular feature. However, earlier employing a BDFL delegate was the last resort and it was done for those aspects of the ecosystem that Guido didn't want to get involved in. With the new governance model, this has become the first resort. Barry Warsaw, said, “...we don't want to make those decisions if there are people in the community who are better equipped to do that. That's what we want to do, we want to allow other people to become engaged with shaping where Python is going to go in the next 25 years.” Hiring a Project Manager to help transition from Python 2 to 3 The countdown for Python 2 has started and it will not be maintained past 2019. The steering council has plans for hiring a Project Manager to help them better manage the sunset of Python 2. The PM will also have the responsibility of looking into minor details as well, for instance, in the documentation, there is mention of Python 2 and 3. These instances will need to be updated, as from 2020 we will only have Python and eventually developers will not have to care about the major numbers. For the systems that haven't migrated, there will be commercial vendors offering support beyond 2020. There will also be options for business-critical systems, but it will take time, shared Willing. One of the responsibilities that the PM role will take care of will be looking into the various best practices that other companies have followed for the migration and help others to easily migrate. Willing said, “Back in a couple of years, Instagram did a great keynote about how they were moving things from 2 to 3. I think one of the things that we want a PM to help us in this transition is to really take those best practices that we're learning from large companies who have found the business case to transition to make it easier.” Status of CPython issue tracking migration from Roundup to GitHub All PSF’s projects including CPython have moved to GitHub, but the issue tracking for CPython is still done through Roundup. Marieta Wijaya, a Platform Engineer at Zapier and a core python developer, wrote PEP 581 that proposes using GitHub for its issue tracking. Barry Warsaw has taken the initial steps and split the PEP into PEP 581 and 588. While PEP 581 gives the rationale and background, PEP 588 gives a detailed plan of how the migration will take place. The council has requested the PSF to hire a PM to take the responsibilities of the migration. Brett Cannon adds, “...with even the PSF about potentially trying to have a PM sort of role to help handle the migration because we realize that if we go forward with this the migration of those issues are going to be critical and we don't want any problems.” The features or improvements Python Packaging Workgroup should now focus on The Python Packaging Workgroup supports the efforts taken for improving and maintaining the packaging ecosystem in Python by fundraising and distributing this fund among different efforts. The efforts this workgroup supports include PyPI, pip, packaging.python.org, setuptools, and cross-project efforts. Currently, the workgroup is supporting the Warehouse project, which is a new implementation of PyPI aiming to solve the issues PyPI users face. Last year, the workgroup came out with the Warehouse code base and in March this year they have laid out work for the next set of improvements which will be around security and accessibility. When Coghlan was asked about what are the next steps now, he shared that they are looking into improving the overall publisher experience. He adds, that though there have been improvements in the consumer experience, very fewer efforts have been put on improving the publisher side. Publisher-side releases are becoming complicated, people want to upload source distributions with multiple wheels for different platforms and different Python versions. Currently, the packaging process is not that flexible. “at the moment the packing index is kind of this instant publish thing like you push it up and it's done...we'd really like to be able to offer a staging area where people can put up all the artifacts for release, make sure everything's in order, and then once they're happy with the release, push button and have it go publish.” These were some of the highlights from the discussion about the changes in the Python governance structure. You can watch the full discussion on YouTube: https://www.youtube.com/watch?v=8dDp-UHBJ_A&feature=youtu.be&t=379 Creators of Python, Java, C#, and Perl discuss the evolution and future of programming language design at PuPPy Mozilla introduces Pyodide, a Python data science stack compiled to WebAssembly RStudio 1.2 releases with improved testing and support for Python chunks, R scripts, and much more!
Read more
  • 0
  • 0
  • 3619
article-image-attackers-wiped-many-github-gitlab-and-bitbucket-repos-with-compromised-valid-credentials-leaving-behind-a-ransom-note
Savia Lobo
07 May 2019
5 min read
Save for later

Attackers wiped many GitHub, GitLab, and Bitbucket repos with ‘compromised’ valid credentials leaving behind a ransom note

Savia Lobo
07 May 2019
5 min read
Last week, Git repositories were hit by a suspicious activity where attackers targeted GitHub, GitLab, and Bitbucket users, wiping code and commits from multiple repositories. The surprising fact is that attackers used valid credentials, i.e. a password or personal access token to break into these repositories. Not only did they sweep the entire repository, but they also left a ransom note demanding 0.1 Bitcoin (BTC). On May 3, GitLab’s Director of Security, Kathy Wang, said, “We identified the source based on a support ticket filed by Stefan Gabos yesterday, and immediately began investigating the issue. We have identified affected user accounts and all of those users have been notified. As a result of our investigation, we have strong evidence that the compromised accounts have account passwords being stored in plaintext on deployment of a related repository.” According to GitLab’s official post, “All total, 131 users and 163 repositories were, at a minimum, accessed by the attacker. Affected accounts were temporarily disabled, and the owners were notified.” This incident first took place on May 2, 2019 at around 10 pm GMT when GitLab received the first report of a repository being wiped off with one commit named ‘WARNING’, which contained a single file containing the ransom note asking the targets to transfer 0.1 BTC (approx. $568) to the attacker’s Bitcoin address, if they want to get their data back. If they failed to transfer the amount, the targets were threatened that their code would be hosted as public. Here’s the ransom note that was left behind: “To recover your lost data and avoid leaking it: Send us 0.1 Bitcoin (BTC) to our Bitcoin address 1ES14c7qLb5CYhLMUekctxLgc1FV2Ti9DA and contact us by Email at admin@gitsbackup.com with your Git login and a Proof of Payment. If you are unsure if we have your data, contact us and we will send you a proof. Your code is downloaded and backed up on our servers. If we dont receive your payment in the next 10 Days, we will make your code public or use them otherwise.” “The targets who had their repos compromised use multiple Git-repository management platforms, with the only other connection between the reports besides Git being that the victims were using the cross-platform SourceTree free Git client”, The Bleeping Computer reports. GitLab, however, commented that they have notified the affected GitLab users and are working to resolve the issue soon. According to BitcoinAbuse.com, a website that tracks Bitcoin addresses used for suspicious activity, there have been 27 abuse reports with the first report filed on May 2. “When searching for it on GitHub we found 392 impacted repositories which got all their commits and code wiped using the 'gitbackup' account which joined the platform seven years ago, on January 25, 2012. Despite that, none of the victims have paid the ransom the hackers have asked for, seeing that the Bitcoin address received only 0.00052525 BTC on May 3 via a single transaction, which is the equivalent of roughly $2.99”, Bleeping Computer mentions. A GitHub spokesperson told the Bleeping Computers, “GitHub has been thoroughly investigating these reports, together with the security teams of other affected companies, and has found no evidence GitHub.com or its authentication systems have been compromised. At this time, it appears that account credentials of some of our users have been compromised as a result of unknown third-party exposures.” Team GitLab has further recommended all GitLab users to enable two-factor authentication and use SSH keys to strengthen their GitLab account. Read Also: Liz Fong-Jones on how to secure SSH with Two Factor Authentication (2FA) One of the StackExchange users said, “I also have 2FA enabled, and never got a text message indicating they had a successful brute login.” One StackExchange user received a response from Atlassian, the company behind Bitbucket and the cross-platform free Git client SourceTree, "Within the past few hours, we detected and blocked an attempt — from a suspicious IP address — to log in with your Atlassian account. We believe that someone used a list of login details stolen from third-party services in an attempt to access multiple accounts." Bitbucket users impacted by this breach, received an email stating, “We are in the process of restoring your repository and expect it to be restored within the next 24 hours. We believe that this was part of a broader attack against several git hosting services, where repository contents were deleted and replaced with a note demanding the payment of ransom. We have not detected any other compromise of Bitbucket. We have proactively reset passwords for those compromised accounts to prevent further malicious activity. We will also work with law enforcement in any investigation that they pursue. We encourage you and your team members to reset all other passwords associated with your Bitbucket account. In addition, we recommend enabling 2FA on your Bitbucket account.” According to Stefen Gabos’ thread on StackExchange Security forum, he mentions that the hacker does not actually delete, but merely alters Git commit headers. So there are chances that code commits can be recovered, in some cases. “All evidence suggests that the hacker has scanned the entire internet for Git config files, extracted credentials, and then used these logins to access and ransom accounts at Git hosting services”, ZDNet reports. https://twitter.com/bad_packets/status/1124429828680085504 To know more about this news and further updates visit GitLab’s official website. DockerHub database breach exposes 190K customer data including tokens for GitHub and Bitbucket repositories Facebook confessed another data breach; says it “unintentionally uploaded” 1.5 million email contacts without consent Understanding the cost of a cybersecurity attack: The losses organizations face
Read more
  • 0
  • 0
  • 4248

article-image-cross-validation-strategies-for-time-series-forecasting-tutorial
Packt Editorial Staff
06 May 2019
12 min read
Save for later

Cross-Validation strategies for Time Series forecasting [Tutorial]

Packt Editorial Staff
06 May 2019
12 min read
Time series modeling and forecasting are tricky and challenging. The i.i.d (identically distributed independence) assumption does not hold well to time series data. There is an implicit dependence on previous observations and at the same time, a data leakage from response variables to lag variables is more likely to occur in addition to inherent non-stationarity in the data space. By non-stationarity, we mean flickering changes of observed statistics such as mean and variance. It even gets trickier when taking inherent nonlinearity into consideration. Cross-validation is a well-established methodology for choosing the best model by tuning hyper-parameters or performing feature selection. There are a plethora of strategies for implementing optimal cross-validation. K-fold cross-validation is a time-proven example of such techniques. However, it is not robust in handling time series forecasting issues due to the nature of the data as explained above. In this tutorial, we shall explore two more techniques for performing cross-validation; time series split cross-validation and blocked cross-validation, which is carefully adapted to solve issues encountered in time series forecasting. We shall use Python 3.5, SciKit Learn, Matplotlib, Numpy, and Pandas. By the end of this tutorial you will have explored the following topics: Time Series Split Cross-Validation Blocked Cross-Validation Grid Search Cross-Validation Loss Function Elastic Net Regression Cross-Validation Image Source: scikit-learn.org First, the data set is split into a training and testing set. The testing set is preserved for evaluating the best model optimized by cross-validation. In k-fold cross-validation, the training set is further split into k folds aka partitions. During each iteration of the cross-validation, one fold is held as a validation set and the remaining k - 1 folds are used for training. This allows us to make the best use of the data available without annihilation. It also allows us to avoid biasing the model towards patterns that may be overly represented in a given fold. Then the error obtained on all folds is averaged and the standard deviation is calculated. One usually performs cross-validation to find out which settings give the minimum error before training a final model using these elected settings on the complete training set. Flavors of k-fold cross-validations exist, for example, leave-one-out and nested cross-validation. However, these may be the topic of another tutorial. Grid Search Cross-Validation One idea to fine-tune the hyper-parameters is to randomly guess the values for model parameters and apply cross-validation to see if they work. This is infeasible as there may be exponential combinations of such parameters. This approach is also called Random Search in the literature. Grid search works by exhaustively searching the possible combinations of the model’s parameters, but it makes use of the loss function to guide the selection of the values to be tried at each iteration. That is solving a minimization optimization problem. However, in SciKit Learn it explicitly tries all the possible combination which makes it computationally expensive. When cross-validation is used in the inner loop of the grid search, it is called grid search cross-validation. Hence, the optimization objective becomes minimizing the average loss obtained on the k folds. R2 Loss Function Choosing the loss function has a very high impact on model performance and convergence. In this tutorial, I would like to introduce to you a loss function, most commonly used in regression tasks. R2 loss works by calculating correlation coefficients between the ground truth target values and the response output from the model. The formula is, however, slightly modified so that the range of the function is in the open interval [+1, -∞]. Hence, +1 indicates maximum positive correlation and negative values indicate the opposite. Thus, all the errors obtained in this tutorial should be interpreted as desirable if their value is close to +1. It is worth mentioning that we could have chosen a different loss function such as L1-norm or L2-norm. I would encourage you to try the ideas discussed in this tutorial using other loss functions and observe the difference. Elastic Net Regression This also goes in the literature by the name elastic net regularization. Regularization is a very robust technique to avoid overfitting by penalizing large weights or in other words it alters the objective function by emphasizing the errors caused by memorizing the training set. Vanilla linear regression can be tricked into learning the parameters that perform very well on the training set, but yet fail to generalize for unseen new samples. Both L1-regularization and L2-regularization were incorporated to resolve overfitting and are known in the literature as Lasso and Ridge regression respectively. Due to the critique of both Lasso and Ridge regression, Elastic Net regression was introduced to mix the two models. As a result, some variables’ coefficients are set to zero as per L1-norm and some others are penalized or shrank as per the L2-norm. This model combines the best from both worlds and the result is a stable, robust, and a sparse model. As a consequence, there are more parameters to be fine-tuned. That’s why this is a good example to demonstrate the power of cross-validation. Crypto Data Set I have obtained ETHereum/USD exchange prices for the year 2019 from cryptodatadownload.com which you can get for free from the website or by running the following command: $ wget http://www.cryptodatadownload.com/cdd/Gemini_ETHUSD_d.csv Now that you have the CSV file you can import it to Python using Pandas. The daily close price is used as both regressor and response variables. In this setup, I have used a lag of 64 days for regressors and a target of 8 days for responses. That is, given the past 64 days closing prices forecast the next 8 days. Then the resulting nan rows at the tail are dropped as a way to handle missing values. df = pd.read_csv('./Gemini_ETHUSD_d.csv', skiprows=1) for i in range(1, STEPS): col_name = 'd{}'.format(i) df[col_name] = df['d0'].shift(periods=-1 * i) df = df.dropna() Next, we split the data frame into two one for the regressors and the other for the responses. And then split both into two one for training and the other for testing. X = df.iloc[:, :TRAIN_STEPS] y = df.iloc[:, TRAIN_STEPS:] X_train = X.iloc[:SPLIT_IDX, :] y_train = y.iloc[:SPLIT_IDX, :] X_test = X.iloc[SPLIT_IDX:, :] y_test = y.iloc[SPLIT_IDX:, :] Model Design Let’s define a method that creates an elastic net model from sci-kit learn and since we are going to forecast more than one future time step, let’s use a multi-output regressor wrapper that trains a separate model for each target time step. However, this introduces more demand for computation resources. def build_model(_alpha, _l1_ratio): estimator = ElasticNet( alpha=_alpha, l1_ratio=_l1_ratio, fit_intercept=True, normalize=False, precompute=False, max_iter=16, copy_X=True, tol=0.1, warm_start=False, positive=False, random_state=None, selection='random' ) return MultiOutputRegressor(estimator, n_jobs=4) Blocked and Time Series Splits Cross-Validation The best way to grasp the intuition behind blocked and time series splits is by visualizing them. The three split methods are depicted in the above diagram. The horizontal axis is the training set size while the vertical axis represents the cross-validation iterations. The folds used for training are depicted in blue and the folds used for validation are depicted in orange. You can intuitively interpret the horizontal axis as time progression line since we haven’t shuffled the dataset and maintained the chronological order. The idea for time series splits is to divide the training set into two folds at each iteration on condition that the validation set is always ahead of the training split. At the first iteration, one trains the candidate model on the closing prices from January to March and validates on April’s data, and for the next iteration, train on data from January to April, and validate on May’s data, and so on to the end of the training set. This way dependence is respected. However, this may introduce leakage from future data to the model. The model will observe future patterns to forecast and try to memorize them. That’s why blocked cross-validation was introduced.  It works by adding margins at two positions. The first is between the training and validation folds in order to prevent the model from observing lag values which are used twice, once as a regressor and another as a response. The second is between the folds used at each iteration in order to prevent the model from memorizing patterns from an iteration to the next. Implementing k-fold cross-validation using sci-kit learn is pretty straightforward, but in the following lines of code, we pass the k-fold splitter explicitly as we will develop the idea further in order to implement other kinds of cross-validation. model = build_model(_alpha=1.0, _l1_ratio=0.3) kfcv = KFold(n_splits=5) scores = cross_val_score(model, X_train, y_train, cv=kfcv, scoring=r2) print("Loss: {0:.3f} (+/- {1:.3f})".format(scores.mean(), scores.std())) This outputs: Loss: -103.076 (+/- 205.979) The same applies to time series splitter as follows: model = build_model(_alpha=1.0, _l1_ratio=0.3) tscv = TimeSeriesSplit(n_splits=5) scores = cross_val_score(model, X_train, y_train, cv=tscv, scoring=r2) print("Loss: {0:.3f} (+/- {1:.3f})".format(scores.mean(), scores.std())) This outputs: Loss: -9.799 (+/- 19.292) Sci-kit learn gives us the luxury to define any new types of splitters as long as we abide by its splitter API and inherit from the base splitter. class BlockingTimeSeriesSplit(): def __init__(self, n_splits): self.n_splits = n_splits def get_n_splits(self, X, y, groups): return self.n_splits def split(self, X, y=None, groups=None): n_samples = len(X) k_fold_size = n_samples // self.n_splits indices = np.arange(n_samples) margin = 0 for i in range(self.n_splits): start = i * k_fold_size stop = start + k_fold_size mid = int(0.8 * (stop - start)) + start yield indices[start: mid], indices[mid + margin: stop] Then we can use it exactly the same way like before. model = build_model(_alpha=1.0, _l1_ratio=0.3) btscv = BlockingTimeSeriesSplit(n_splits=5) scores = cross_val_score(model, X_train, y_train, cv=btscv, scoring=r2) print("Loss: {0:.3f} (+/- {1:.3f})".format(scores.mean(), scores.std())) This outputs: Loss: -15.527 (+/- 27.488) Please notice how the loss is different among the different types of splitters. In order to interpret the results correctly, let’s put it to test by using grid search cross-validation to find the optimal values for both regularization parameter alpha and -ratio that controls how much -norm contributes to the regularization. It follows that -norm contributes 1 - . params = { 'estimator__alpha':(0.1, 0.3, 0.5, 0.7, 0.9), 'estimator__l1_ratio':(0.1, 0.3, 0.5, 0.7, 0.9) } for i in range(100): model = build_model(_alpha=1.0, _l1_ratio=0.3) finder = GridSearchCV( estimator=model, param_grid=params, scoring=r2, fit_params=None, n_jobs=None, iid=False, refit=False, cv=kfcv, # change this to the splitter subject to test verbose=1, pre_dispatch=8, error_score=-999, return_train_score=True ) finder.fit(X_train, y_train) best_params = finder.best_params_ Experimental Results K-Fold Cross-Validation Optimal Parameters Grid-search cross-validation was run 100 times in order to objectively measure the consistency of the results obtained using each splitter. This way we can evaluate the effectiveness and robustness of the cross-validation method on time series forecasting. As for the k-fold cross-validation, the parameters suggested were almost uniform. That is, it did not really help us in discriminating the optimal parameters since all were equally good or bad. Time Series Split Cross-Validation Optimal Parameters Blocked Cross-Validation Optimal Parameters However, in both the cases of time series split cross-validation and blocked cross-validation, we have obtained a clear indication of the optimal values for both parameters. In case of blocked cross-validation, the results were even more discriminative as the blue bar indicates the dominance of -ratio optimal value of 0.1. Ground Truth vs Forecasting After having obtained the optimal values for our model parameters, we can train the model and evaluate it on the testing set. The results, as depicted in the plot above, indicate smooth capture of the trend and minimum error rate. # optimal model model = build_model(_alpha=0.1, _l1_ratio=0.1) # train model model.fit(X_train, y_train) # test score y_predicted = model.predict(X_test) score = r2_score(y_test, y_predicted, multioutput='uniform_average') print("Test Loss: {0:.3f}".format(score)) The output is: Test Loss: 0.925 Ideas for the Curious In this tutorial, we have demonstrated the power of using the right cross-validation strategy for time-series forecasting. The beauty of machine learning is endless. Here you’re a few ideas to try out and experiment on your own: Try using a different more volatile data set Try using different lag and target length instead of 64 and 8 days each. Try different regression models Try different loss functions Try RNN models using Keras Try increasing or decreasing the blocked splits margins Try a different value for k in cross-validation References Jeff Racine,Consistent cross-validatory model-selection for dependent data: hv-block cross-validation,Journal of Econometrics,Volume 99, Issue 1,2000,Pages 39-61,ISSN 0304-4076. Dabbs, Beau & Junker, Brian. (2016). Comparison of Cross-Validation Methods for Stochastic Block Models. Marcos Lopez de Prado, 2018, Advances in Financial Machine Learning (1st ed.), Wiley Publishing. Doctor, Grado DE et al. “New approaches in time series forecasting: methods, software, and evaluation procedures.” (2013). Learn More Seize the chance to learn more about time series forecasting techniques, machine learning, trading strategies, and algorithmic trading on my step by step online video course: Hands-on Machine Learning for Algorithmic Trading Bots with Python on PacktPub. Author Bio Mustafa Qamar-ud-Din is a machine learning engineer with over 10 years of experience in the software development industry engaged with startups on solving problems in various domains; e-commerce applications, recommender systems, biometric identity control, and event management. Time series modeling: What is it, Why it matters and How it’s used Implementing a simple Time Series Data Analysis in R Training RNNs for Time Series Forecasting
Read more
  • 0
  • 0
  • 44245

article-image-jupyterhub-1-0-releases-with-named-servers-support-for-tls-encryption-and-more
Sugandha Lahoti
06 May 2019
4 min read
Save for later

JupyterHub 1.0 releases with named servers, support for TLS encryption and more

Sugandha Lahoti
06 May 2019
4 min read
JupyterHub 1.0 was released last week as the first major update since 2015. JupyterHub allows multiple users to use Jupyter notebook. JupyterHub 1.0 comes with UI support for managing named servers, and TLS encryption and authentication support, among others. What’s new in JupyterHub 1.0? UI for named servers JupyterHub 1.0 comes with full UI support for managing named servers. Named servers allow each Jupyterhub user to have access to more than one named server. JupyterHub 1.0 introduces a new UI for managing these servers. Users can now create/start/stop/delete their servers from the hub home page. Source: Jupyter blog TLS encryption and authentication JupyterHub 1.0 supports TLS encryption and authentication of all internal communication. Spawners must implement .move_certs method to make certificates available to the notebook server if it is not local to the Hub. Currently, local spawners and DockerSpawner support internal ssl. Checking and refreshing authentication JupyterHub. 1.0 introduces three new configurations to refresh or expire authentication information. c.Authenticator.auth_refresh_age allows authentication to expire after a number of seconds. c.Authenticator.refresh_pre_spawn forces a refresh of authentication prior to spawning a server, effectively requiring a user to have up-to-date authentication when they start their server. Authenticator.refresh_auth defines what it means to refresh authentication and can be customized by Authenticator implementations. Other changes A new API is added in JupyterHub 1.0 for registering user activity. Activity is now tracked by pushing it to the Hub from user servers instead of polling the proxy API. Dynamic options_form callables may now return an empty string which will result in no options form being rendered. Spawner.user_options is persisted to the database to be re-used so that a server spawned once via the form can be re-spawned via the API with the same options. c.PAMAuthenticator.pam_normalize_username, option is added for round-tripping usernames through PAM to retrieve the normalized form. c.JupyterHub.named_server_limit_per_user configuration is added to limit the number of named servers each user can have. The default is 0, for no limit. API requests to HubAuthenticated services (e.g. single-user servers) may pass a token in the Authorization header, matching authentication with the Hub API itself. Authenticator.is_admin(handler, authentication) method and Authenticator.admin_groups configuration is added for automatically determining that a member of a group should be considered an admin. These are just a select few updates. For the full list of new features and improvements in JupyterHub 1.0, visit the changelog. You can upgrade jupyterhub with conda or pip: conda install -c conda-forge jupyterhub==1.0.* pip install --upgrade jupyterhub==1.0.* Users were quite excited about the release. Here are some comments from a Hacker News thread. “This is really cool and I’m impressed by the jupyter team. My favorite part is that it’s such a good product that beats the commercial products because it’s hard to figure out, I think, commercial models that support this wide range of collaborators (people who view once a month to people who author every day).” “Congratulations! JupyterHub is a great project with high-quality code and docs. Looking forward to trying the named servers feature as I run a JupyterHub instance that spawns servers inside containers based on a single image which inevitably tends to grow as I add libraries. Being able to manage multiple servers should allow me to split the image into smaller specialized images.” Introducing Jupytext: Jupyter notebooks as Markdown documents, Julia, Python or R scripts How everyone at Netflix uses Jupyter notebooks from data scientists, machine learning engineers, to data analysts. 10 reasons why data scientists love Jupyter notebooks
Read more
  • 0
  • 0
  • 4920
article-image-f8-pytorch-announcements-pytorch-1-1-releases-with-new-ai-toolsopen-sourcing-botorch-and-ax-and-more
Bhagyashree R
03 May 2019
4 min read
Save for later

F8 PyTorch announcements: PyTorch 1.1 releases with new AI tools, open sourcing BoTorch and Ax, and more

Bhagyashree R
03 May 2019
4 min read
Despite Facebook’s frequent appearance in the news for all the wrong reasons, we cannot deny that its open source contributions to AI have been its one redeeming quality. At its F8 annual developer conference showcasing its exceptional AI prowess, Facebook shared how the production-ready PyTorch 1.0 is being adopted by the community and also the release of PyTorch 1.1. Facebook introduced PyTorch in 2017, and since then it has been well-received by developers. It partnered with the AI community for further development in PyTorch and released the stable version last year in December. Along with optimizing and fixing other parts of PyTorch, the team introduced Just-in-time compilation for production support that allows seamless transitions between eager mode and graph mode. PyTorch 1.0 in leading businesses, communities, and universities Facebook is leveraging end-to-end workflows of PyTorch 1.0 for building and deploying translation and NLP at large scale. These NLP systems are delivering a staggering 6 billion translations for applications such as Messenger. PyTorch has also enabled Facebook to quickly iterate their ML systems. It has helped them accelerate their research-to-production cycle. Other leading organizations and businesses are also now using PyTorch for speeding up the development of AI features. Airbnb’s Smart Reply feature is backed by PyTorch libraries and APIs for conversational AI. ATOM (Accelerating Therapeutics for Opportunities in Medicine) has come up with a variational autoencoder that represents diverse chemical structures and designs new drug candidates. Microsoft has built large-scale distributed language models that are now in production in offerings such as Cognitive Services. PyTorch 1.1 releases with new model understanding and visualization tools Along with showcasing how the production-ready version is being accepted by the community, the PyTorch team further announced the release of PyTorch 1.1. This release focuses on improved performance, brings new model understanding and visualization tools for improved usability, and more. Following are some of the key feature PyTorch 1.1 comes with: Support for TensorBoard: TensorBoard, a suite of visualization tools, is now natively supported in PyTorch. You can use it through the  “from torch.utils.tensorboard import SummaryWriter” command. Improved JIT compiler: Along with some bug fixes, the team has expanded capabilities in TorchScript such as support for dictionaries, user classes, and attributes. Introducing new APIs: New APIs are introduced to support Boolean tensors and custom recurrent neural networks. Distributed training: This release comes with improved performance for common models such as CNNs. Multi-device modules support and the ability to split models across GPUs while still using Distributed Data Parallel is added. Ax, BoTorch, and more: Open source tools for Machine Learning engineers Facebook announced that it is open sourcing two new tools, Ax and BoTorch that are aimed at solving large scale exploration problems both in research and production environment. Built on top of PyTorch, BoTorch leverages its features such as auto-differentiation, massive parallelism, and deep learning to help in researches related Bayesian optimization. Ax is a general purpose ML platform for managing adaptive experiments. Both Ax and BoTorch use probabilistic models that efficiently use data and meaningfully quantify the costs and benefits of exploring new regions of problem space. Facebook has also open sourced PyTorch-BigGraph (PBG), a tool that makes it easier and faster to produce graph embeddings for extremely large graphs with billions of entities and trillions of edges. PBG comes with support for sharding and negative sampling and also offers sample use cases based on Wikidata embedding. As a result of its collaboration with Google, AI Platform Notebooks, a new histed JupyterLab service from Google Cloud Platform, now comes preinstalled with PyTorch. It also comes integrated with other GCP services such as BigQuery, Cloud Dataproc, Cloud Dataflow, and AI Factory. The broader PyTorch community has also come up with some impressive open source tools. BigGAN-Torch is basically a full reimplementation of PyTorch that uses gradient accumulation to provide the benefits of big batches by only using a few GPUs. GeomLoss is an API written in Python that defines PyTorch layers for geometric loss functions between sampled measures, images, and volumes. It provides efficient GPU implementations for Kernel norms, Hausdorff divergences, and unbiased Sinkhorn divergences. PyTorch Geometric is a geometric deep learning extension library for PyTorch consisting of various methods for deep learning on graphs and other irregular structures. Read the official announcement on Facebook’s AI  blog. Facebook open-sources F14 algorithm for faster and memory-efficient hash tables “Is it actually possible to have a free and fair election ever again?,” Pulitzer finalist, Carole Cadwalladr on Facebook’s role in Brexit F8 Developer Conference Highlights: Redesigned FB5 app, Messenger update, new Oculus Quest and Rift S, Instagram shops, and more
Read more
  • 0
  • 0
  • 3323

article-image-sleep-loss-cuts-developers-productivity-in-half-research-finds
Vincy Davis
03 May 2019
3 min read
Save for later

All coding and no sleep makes Jack/Jill a dull developer, research confirms

Vincy Davis
03 May 2019
3 min read
In recent years, the software engineering community has been interested in factors related to human habits that can play a role in increasing developers' productivity. The researchers- D. Fucci from HITeC and the University of Hamburg, G. Scanniello and S. Romano from DiMIE - University and N. Juristo from Technical University of Madrid have published a paper “Need for Sleep: the Impact of a Night of Sleep Deprivation on Novice Developers’ Performance” that investigates how sleep deprivation can impact developers' productivity. What was the experiment? The researchers performed a quasi experiment with 45 undergraduate students in Computer Science at the University of Basilicata in Italy. The participants were asked to work on a programming task which required them to use the popular agile practice of test-first development (TFD). The students were divided into two groups - The treatment group where 23 students were asked to skip their sleep the night before the experiment and the control group where the remaining students slept the night before the experiment. The conceptual model and the operationalization of the constructs investigated is as shown below. Image source: Research paper Outcome of the Experiment The result of the experiment indicated that sleep deprivation has a negative effect on the capacity of software developers to produce a software solution that meets given requirements. In particular, novice developers who forewent one night of sleep, wrote code which was approximately 50% more likely not to fulfill the functional requirements with respect to the code produced by developers under normal sleep condition. Another observation was that sleep deprivation decreased developers' productivity with the development task and hindered their ability to apply the test-first development (TFD) practice. The researchers also found that sleep-deprived novice developers had to make more fixes to syntactic mistakes in the source code. As an aftereffect of this result paper, experienced developers are recollecting their earlier sleep deprived programming days. Some are even regretting them. https://twitter.com/zhenghaooo/status/1121937715413434369 Recently the Chinese ‘996’ work routine has come into picture, wherein tech companies are expecting their employees to work from 9 am to 9 pm, 6 days a week, leading to 60+ hours of work per week. This kind of work culture will devoid these developers of any work-life balance. This will also encourage the habit of skipping sleep. Thus decreasing developers productivity. A user on Reddit declares sleep as the key to being a productive coder and not burning out. Another user added, “There's a culture in university computer science around programming for 30+ hours straight (hackathons). I've participated and pulled off some pretty cool things in 48 hours of feverish keyboard whacking and near-constant swearing, but I'd rather stab myself repeatedly with a thumbtack than repeat that experience.” It’s high time that companies focus more on the ‘quality’ of work than insisting developers to work for long hours, which will in turn reduce their productivity. It is clear from the result of this research paper that no sleep in a night, can certainly affect one’s quality of work. To know more about the experiment, head over to the research paper. Microsoft and GitHub employees come together to stand with the 996.ICU repository Jack Ma defends the extreme “996 work culture” in Chinese tech firms Dorsey meets Trump privately to discuss how to make public conversation “healthier and more civil” on Twitter
Read more
  • 0
  • 0
  • 7657