How do you teach a computer to type without a keyboard? Artificial intelligence at the SSGN

3 July 2018

Article written by Jeanine Schoonemann and Stijn Kuijpers

It’s a May morning at Nijmegen City Comprehensive School. There are 50 students, 7 laptops with webcams and 26 letters. The students are set a task: invent your own sign language and teach the computer to recognise your signs so that you can type without a keyboard.

Socially engaged: education for the next generation

Cmotions is a company that believes passionately in being socially engaged. This is something we are doing through our “Who Cares?” programme. As part of the topic of “education for the next generation”, we gave a two-day workshop at Nijmegen City Comprehensive School (SSgN). The workshop took place during project week for some 50 pre-university secondary school (VWO) students. In the workshop, we wanted to introduce a group of teenagers in an interactive way to our world of data, analytics and artificial intelligence.

Artificial intelligence and your own sign language

We kicked off the workshop with a short presentation on how and why companies collect your personal data and how they use artificial intelligence. Then there was a practical part to really get their teeth into the subject.

The techies & the creatives

For the task, we split the students into seven groups. The “techies” (students taking ICT as a subject) and the “creatives” worked together on a unique sign alphabet and an algorithm that can recognise those signs on the computer via the webcam. The students were given the goal to input a text via webcam as quickly and accurately as possible in a final battle on day 2. Let the games begin!

The fusion of the groups to make the alphabet

To utilise their qualities and interests as much as possible, the techies delved deeper into the world of object recognition and Python. Meanwhile, the creatives thought up a sign for every letter of the alphabet. After the groups “fused”, the techies knew more about the requirements that the photos needed to satisfy in order to train the model. The group could then continue together with the 26 letters. Up to 30 photos were needed for each letter.

Success factors for the sign language

A successful sign language was already standing out from the crowd. The following factors were important:

• For letters that recur a lot, it is a good idea to have a simple (and therefore quick) sign.
• Signs that look completely different to you may not always be completely different to a computer.
• Shadow is tricky for photos, but not necessarily a problem in itself.
• Not everyone is equally flexible when it comes to twisting their arms, wrists and fingers!

Luckily, after the first day, the laptops had a good 36 hours in which to turn the photos into an intelligent algorithm. An algorithm that can identify a unique sign language and until recently didn’t even exist.

Possibilities vs. privacy: the debate

On the morning of day two, we got cracking bright and early with a lively debate on the advantages and disadvantages of the use of data and artificial intelligence. Adopting a House of Commons format, the students successfully debated in detail the subjects of social media platforms & privacy, “smart” camera surveillance by the government, autonomous weapons and more. We were positively surprised by the quality and creativity of the arguments used by those arguing both for and against the motions.

The battle: which sign language was recognised the best

And finally the time had come for: the battle! In the practice rounds it soon became clear that not all letters would be as easily recognised as each other, but that every group was able to at least “type” a handful of letters consistently and quickly. The two top teams finished head and shoulders ahead of the rest. They typed a large part of the final text in 5 minutes without a single mistake. In the end, the winning team was the one that gave the most thought to the signs for frequently recurring letters and had the best collaboration between techies and creatives. Well done!

A successful workshop thanks to enthusiastic collaboration between business and education

We hugely enjoyed preparing and delivering this workshop for an unfamiliar audience. It wasn’t as easy as you’d think! Fortunately, the students were enthusiastic and the school administration were very positive about this collaboration between business and education. We hope that through this workshop we can contribute both now and in the future to “education for the next generation” because it certainly feels like the start of something!

View the photos of the 2-day workshop here: