>> Education, this is rtp is is rtp one 18.

>> He's a North Carolina native and the director of marketing communications for the research Triangle Foundation.

You may have seen or behind the scenes, this is the first time up on this stage, your friend and hers Morgan what's the try not to trip area.

Hi everybody welcome back haha this is really exciting.

So like we said I'm here to talk to you a little bit about the work we do at the research Triangle Foundation.

They are the.

>> Stewart's and owners of research in the park including immunity is like the building that we're sitting in right now.

And I'm here to talk to you about are rewriting project which we undertook in 2019.

Now as you can imagine a brand like ours with rtp being founded in 1959 has quite a lot of emotion around it for people here is your hand if you're from the triangle or lived here long enough that you feel like you're from here okay so I was actually born at UNC hospital so I've been a tarheel since the first day I was on Earth and I am really proud to you be from here.

And those of you who live here probably here because of the triangle right you heard the word million times.

But what is really mean and what does it have to do with branding lot.

A brand isn't just a logo, it's all these things it's all the things that lie just talked about.

Thanks for setting me up.

It's how you engage with people, but it's also how you make connections with people.

It's how you ensure that people want to be associated with you think about why you moved here if you moved here think about why you work in rtp think about what that means to be part of this university ecosystem that's part of what.

The story Tony we do every day, but it's also your logo right.

It's how you talk about it your mission statement all those good things.

Give an example that some of you have probably seen this brand before.

So at the Nike swoosh now like I mentioned we underwent a rebranding in 2019.

So right before the pandemic and we realized the brand that we had wasn't working for us.

It's what I want everyone here to take away is that whatever is working for your brand as we need to look for and what you need to stick with.

So when we talk about Nike.

It's not just their logo right it's their slogan just do it.

It's also the brands that they manage that the family of companies that they run.

It's their celebrity influencers it's their endorsements.

It's their advertising inclusive of their audience, everyone from children to different sizes and genders and races of people.

They've also got some super fans.

And they've also got some super super super fans haha.

So.

So when you think about a brand like that so ubiquitous so many audience members to consider in their digital marketing.

What stay the same.

That's what they realize this is a symbol that everybody but he recognizes everybody appreciates and it makes people feel something right.

So they've kept that swoosh even though they've changed their fonts they've changed their coloring a little bit how they treat it.

It's still there's been there since 1970 want to 50 years.

Rtp is the same way now this just goes back to 1996 we've had quite a few iterations of our logo.

One thing that has stayed the same as what worked for us and that's the triangle is that symbol that unites the 3 cities that make up our region.

But it also came to represent a lot more a collaboration between the universities which is represented in 3 colors.

And of course the the interactivity between all of us that makes this such a unique place to live and do business.

So our new logo so that different from the old one and see up there in the top rate.

But we changed was really because of digital marketing what we had was a cool logo, great representation of our geography in our schools that we didn't have was something that would work well with social media and it also didn't work well as we were of all Ving to have several new brands underneath our brand so I mentioned the frontier campus were all sitting FOX yard across the street.

Which is the pilot are over there in stem rtp which is our grantmaking program for K through 12 schools.

So what we really wanted to do was come up with not only this this logo refreshed but also a way that you could easily identify just like everyone in your family might have Brown highs or a volunteer.

That you could easily notice this thing has its own identity, but it's part of our tp so we came up with this secondary badge that we call it you see in the top rate.

And we use that with all of our other entities from near with that enable us to do is create that consistent experience just like every Nike product has that consistent feel every single one of our this is a good example of of digital and social media and our Web site at the bottom left you see the consistency in the colors, how it feels how we talk about it.

We also have celebrity endorsements there in the form of companies investing here in the triangle some amazing.

As I'm sure you all have have read lately even in the midst of a pandemic people are recognizing the value of being here growing a company here moving their families here.

And we have our super fans we have people on social media who are tagging us coming out and visiting people who maybe would never come to the triangle or to rtp before are starting to come here.

And it's been really exciting to see that evolution and are ideal brand structure is what has enabled us to reach all of them.

Thank you that I have for you.

>> You know I had today you're right just just cause he's always alright, Lazio has time for questions about the branding of RDP and how that fits in the content marking raise your hand if you've got a question.

We will get you an answer.

Also we've got our first question over here is I'm mobile there's an fabric first question over here.

>> You said their original marketing wasn't working or their original brand wasn't working, you know what exactly about it wasn't working.

I had you know it's not working.

>> Absolutely so yes, how you know, you're branding isn't working.

So for example you can see our old Logan go back to it.

Too far too far.

So if you've ever looked at your phone and you look in an instagram profile can you imagine trying to fit that thing in that little tiny circle.

It's really really wasn't working so digital marketing requires you to have a lot of different shapes sizes colors, things that you can use in a variety of different platforms backgrounds and all of those amazing channels that Laurie mentioned earlier have to be a lot more adaptable.

>> All right other questions cool, we got one right here.

Coming your way for a question about digital branding.

>> Yeah, I notice that you have the colors of the colleges in the triangle so we have UNC Chapel Hill and NC state yeah like that.

Yes, so those are the 3 universities that founded rtp in the 1950's.

They came together with the governor at the time and several other business leaders too for what was really the first ever economy of ideas is what we call it sorry should stay dry was what's really cool though about our gradient.

If you look between the tea and the P we were able to add the maroon color in that gradient between rtp of NC central eagles.

So we're incorporating yeah.

They're also playing a such a big role as well as Durham Tech and wake Tech in her whole ecosystem of universities in the amazing talented people that are graduating every year and going to work for the amazing companies in the park so we're really excited to be able to do that.

>> All right next question coming here from the back left and then we'll get around front.

>> So Research Triangle Park is trademarked right so how do you.

How do you want people to celebrate rtp without.

I'm just I guess I'm asking that the how do you differentiate that because obviously your logo can be trademark but rtp itself cannot so how did what was the thought on that on that approach of branding everything rtp.

>> We like it haha there's a rtp Marriott I you know there are a lot of companies who write press releases that are over in Morrisville but they put Research Triangle Park in the in the dateline because there is so much value in our name and so we're we're really proud of that that people want to be associated with the park.

Question here up front.

>> I'm curious of metrics a meaningless be tracking numbers right so what's the reach for rtp is that hundreds thousands.

>> Millions last time I checked we get probably is half a million website visits a month more.

I'm looking at my team.

Yeah, we get global reach.

Online our website or social media, our news letter I think we have 10's of thousands of people subscribe to that so we have a really wide audience.

Just like Nike, we have to be mindful of that with everything that we do.

>> All right other questions we've got one here up front believe they follow up coming here.

>> So first of all want to applaud you for your work, I think I love what you did.

And to the fact of I knew that was a question about the rtp but rtp really carries a lot of weight in for you to sub ran that and year rather lines was was really good.

Thank you so I really like that we went through similar exercise and done something similar say you come from some of the thought process ease though we went through so we want to applaud you for as rtp is a great place and that's why I'm here.

>> I can't lie, we were inspired by old Navy gap that have anybody has anybody shop there.

So it's it's old maybe dot gap .com.

It's it's really gap taking ownership.

They're they're solid name and then just associating those other brands, it owns with it so we kind of took that same model and applied it to our Park.

>> And then last question are coming for me actually as rtp evolves from a pure office Park in 2.

A place where people live work shop how do you anticipate the brand growing and changing in the future.

>> Thank you wait for plugging that.

We are building the first-ever residential apartments in rtp is boundary right now that the dirt pile.

I mentioned earlier so really excited to have people not only be able to commute to rtp but avoid 40 altogether take are many trails to bike or walk to work from their apartment just down the street.

And as that involves I'm really excited to I have this Foundation of tools that we can use to continue adapting I'm sure a lot just like digital marketing it will continue evolving and we're ready to of all with it right guys.

He's young a big hand for Morgan what's the.

>> All you need to know about RDP but were afraid to ask.

Speaking of if you ask questions you're getting this way you may have seen some of our staff running around that sunscreen if you look like me you need it.

Yeah, I've been outside often it's is I got my 2 that's all I'm saying all right, let's zoom in we're now into our 3rd speaker of the evening, he's a serial startup operators dedicate his career to scaling organizations.

He's currently working with keen please welcome Mike.

>> All tall.

>> All right.

Thank you very much.

Hello, everyone as you just heard my name is Mike all top can you hear me yes, good and I thought I'd throw my picture up 7 times in this talks of his the second you can count along with me.

My is my kind of we're kicking decision systems and if you heard my intro.

I am SAS operator.

So if your BS meter is going off this kayak can't know anything about digital.

There's a marketing because he has no expense, you're pretty much right.

I am not a digital marketing so applaud your alarm and it's well.

It's well tuned.

What I am however is fortunate enough to over the last years got to work with a team of incredible marketing Analytics and marketing investment decision folks who have worked tirelessly over I can actually say decades too help brands predominantly large consumer packaged good brands today, many of which you see on the shelves of target Walmart and the grocery stores that you might go to figure out how to spend their marketing dollars in the most effective way to drive financial contribution to their business and that gets me really excited not only as a technologist who believes that we can use computers to be better be more efficient serve serve.

One another, but also because I've seen the results of of what Keane is able to do.

So what I'm going to do.

In the remaining time here is make an analogy that might tie what I see as a non-market are going on in in the world of marketing as it relates to to digital.

So the number and throughout his 15,000 we could play find S and get an answer what is this number but I was going to give you the answer.

15,000 is than average number of times a human blanks on any given day and why am I talking about blinking.

Well.

If you think about a blink as a decision.

There's this micro moment that our brain has a processor has gone through where it collected lots of data right.

It's it's perhaps measured the wetness of our eye the wind that we're facing the last time we blinked, and it said you know it now is the time next to blink again.

And there's more blinking and over the course of the day we've got this brain of ours that's processing what feels like an intentionally at this point as is processing these decisions and making the choice to blink when I say decision I'm talking about drawing a conclusion.

After consideration after taking in data and one of the things that the team does is we as we help marketers make decisions.

But decision 30 years ago it was actually great to get the marketing timeline because I was going to say about 30 years ago but before the World Wide Web in 1993 before there was a digital marketers decisions were very different, you know there was and that's not to say they weren't creative or challenging.

But different, you know there was your set of copy there is your set of imagery that you may use and then there were very specific channels that you could deploy marketing in there was television that was pranked there was radio it was out of home.

And in-store promotion.

There were very specific channels with a significant supply chain to actually engage in that channel so marketer may actually decide on the copy in the messaging make a purchase order wait some amount of 0 to a 180 or 360 days before it actually landed on TV and then hope the product it got purchased that whole world as you might imagine, has blown up with digital marketing right now in in the in the Old world where you might actually have the time and space to make the decisions that you want to focus on digital has changed that.

Now marketers have to rely on other ways to get the decisions that they need to make made.

And that is where maybe marketers have to learn to blink so what I mean by blink is actually relying on the tools and the abilities of of decision support tool air decision support tools like what keen eyes to actually wrote make the decisions for them and that's not to say that they're not involved in.

In tuning what the decisions may be in that defining their goals, but ultimately if we think about digital.

Digital is this world where now there's all sorts of new channels things like e-commerce site social media sites, many of the logo as we've seen here in the first 2 presentations if you go the next level down marketers now have the option to choose what messaging and what imagery they might deliver to a specific geography to us specific demographic until gdpr and other things all the way down to the specific individual consumer they can make a choice about what they want to say to an individual.

How do you how do you make the right decision about when and where to put your dollar as a marketer to get the right message across.

I think the answer is the machines.

Now this is not a threat that the robots are coming.

But I believe that the right tools better enable by software are what marketers must rely on and are certainly beginning to rely on as we saw in that one of the graphics from earlier that are supporting them and giving them confidence to make the risk and and many of the high pressure situation decisions that they are faced with every day.

Keane has a decision-making platform that you know right out of the box today.

What we enable marketers to do if they're considering where and how they might move digital money from us from online search and or into tiktok they can actually go into the platform and say hey what if and before they ever put a dollar in the market or deal with whatever the reactions are of the market they can use a model that's informed by historical data and continuously updating information empowered by machine learning and AI to actually have a high degree of confidence that before that dollar ever makes it in the market, it's going to meet the goals of their business that's what we believe decision support can be for marketers.

So from my perspective as a non-market or I'm really excited about the advent of how technology solutions are going to support digital marketers not only to meet the scale that they have a are trying that the objectives that they're trying get the right tape, he eyes cost per click etcetera but I actually believe in and would hypothesize that with the advent of the tooling that is coming alongside digital marketing to support the right decisions and the right investment.

Optionality marketers are now entering into into a place where unlike what I think they've had to deal with over the last 30 years of navigating this growing uncertainty and tooling now actually go to revert back to some of that strategic creativity and ingenuity of marketers of past and I believe it's actually this this digital support that that technologies like what he does actually in supporting digital marketing decisions is going to unlock a whole nother level of creativity.

My hope is that with tooling we're going to teach digital marketers how to blink.

>> Good stuff.

There questions about the intersection between technology and marketing raise your hand you can get some sun screen out of it, not gonna lie, it's good stuff.

All righty questions here got one right up front.

>> So if you've got a machine-learning approach to helping workers make these decisions but is the training data look like where does it come from.

>> Yeah that is a very reasonable question.

So one of the things that we talk about with our own clients is moving down the value chain as they're thinking about their dollars so where you might put millions of dollars overseas where you might put 10's or hundreds or thousands of dollars is really the first decision we're really pushing clients to ask and I think it it starts to answer the question you're getting at which is you know, there's there's obviously infinite amounts of data how do you know what the right data is and frankly, it's not necessarily.

The machine learning that's valuable.

It's having a a tool that gives you a view that is data informed into the ecosystem that you're investing in that is the first step and so the machine learning is really teaching is more machine take teaching at this point in such a way that that marketers are using data properly for the decisions they make.

>> Excellent other questions about the intersection of technology and digital marketing.

We've got one that came in on social media using the hashtag rtp one 80 that question was with the ever changing landscape of privacy versus marketing how do you balance that and what sort of opportunities do you see in the future, given the increasing emphasis on user privacy.

>> Yes, from a team perspective we always are asking how to what decision are you making.

And so one of the realities of that is if you might be making is you can't decide what you're going to give to an individual consumer because of the privacy laws, we actually think that's OK and that just becomes a reality or a constraint on the of the decision you're supporting.

And so from from my perspective privacy doesn't prevent marketers from delivering the right marketing strategy to market.

It actually just as a constraint into the way they need to think about going to market and so from our perspective, yes, it at it will have an impact on where you might invest and how you might have data about where you invested, but it should not in particular has the optimizations and this offers get faster should not actually have much of an impact on the ability to put the dollars where there is where the impact is machin me the sales.

Excellent other questions about digital marketing and technology raise your hand.

>> Wow you think of what I've got one because I'm a software developer myself all right and there's a natural tension, I think between soffer development product and marketing both sides kind of have a traditionally a somewhat antagonistic relationship between each other.

How does good marketing enable strong software to thrive in a way that just software by itself might not.

Yeah you got mean, and how does good Mark good software development out of our cutting it off we're working a lot lot of software developers don't think they need marketing like I build a great product people just come find it.

>> You need you need marketing get the word out there.

Yeah, expand the question a little bit to say the beauty of what we think about when we think about whether and where to invest marketing dollars.

Is you can build a model that will show whether you do or don't need to put money towards marketing your product and so and you can actually evaluate all right there.

You have goals that you want to reach their revenue goals that you want to their adoption goals and if the data is not showing that you're doing that and there's probably a mathematical conversation to be had around your software needing to marketing.

>> All right any other questions about software and digital marketing going once going twice last year with big hand like tall.

All right that rain just keeps on rolling here at our TV one 80 up next.

She's a content marketing manager she's also a triangle business Journal 40 under 40 please welcome to sage push or a what >> all right.

Thank you all covid makes a couple more steps have to come into place before grabbing the Mike.

So my name is bashar black and a content marketing manager at night where a software company content governance collaboration.

Look at the learn more about us, but that's not why I'm here today.

And so when I started thinking about what I wanted my topic to before today's talk around community building.

I thought about kind of what I've done in my work.

How I didn't necessarily have a traditional pathway to marketing which I know a lot of people in marketing are in that same boat.

And so I tried to draw on what I've experienced in my career and how that is tied into what I do now in digital marketing as a content marketing manager.

So this is kind of how it started we touched on that a bit today.

We were all kind of connecting prepandemic I know I myself I was going to one to 2 events a week and learning from people meeting people.

You know that's one of the things people really really love about the triangle that is community-oriented and everyone just wants to help and grow together.

And now we're kind of looking like this with senator does were waving at people virtually I've met some of my co-workers in person, but the first time here today which is kind of cool.

It kind of weird but you know that's where we are and so the question kind of poses how do you build community and how do you connect with people when we're all sitting behind a computer.

And let's be honest you probably are having virtual events on one screen and you're working on the other or your listening to a call in one hit phone and then you're also probably talking to your kid or your dog or your partner, whoever's so how do we keep people to engage in give them that interaction that they're really craving.

So I'm going to go into a particular company that I enjoy when I think we've all spent a lot of time with during the pandemic and that I think does a really great job of building online community.

But first I'm going to have been to this line here that just talked a little bit about the different types of digital marketing.

I know we talked earlier about what traditional marketing looks like an.

This is just the kind of give you a visual and now I was open to it.

So netflix I know we've all been netflix seen during the pandemic I have kind of this overall picture here you can see a few pieces of their content there's probably 1, 2, or 3 things that you can pick out that you're really interested in but maybe this isn't enough to make you sign up into Tuesday netflix over hulu or slaying are paramount plus order whatever streaming service is coming out.

And netflix the and so they figured out a way to segment their audiences speak to them directly and helped them build online community.

Inter strong black lead.

Strong black lead is a campaign that netflix runs it speaks specifically to black voices and black stories and speak to me directly as a young black millennial and what it does is they saw within the market that there was a cry for there to be black stories that didn't feature suffering and didn't feature struggle but really showed strong black leads and so they saw that they talked with their audience and not at their audience and answer that.

And so you can see here they have online instagram community of over 7 or exactly 700,000 followers.

So the 700,000 people, including myself that is following with strong black lead is doing and keeping up with what they think that I want to learn about and you can see they're putting here is will at most which is for their lgbtq plus community if you're interested.

And so they're giving me an overview of everything that's coming out for the month.

Things that are either.

Things that came out when I was younger like a lot of fools fall in love but then there's also new things like the Naomi Osaka documentary that's amazing so I'm putting these things on my calendar every month tuning in I'm excited to watch them.

I'm talking with my friends about it, then they're going to be really cool content like this that show's characters as Bridget and I'm sure a lot of you watch I have been watching that during the pandemic this is making me want to watch it even more go back and watch it.

I'm laughing at that funny moment that I didn't think anyone else.

But me and now I'm staring this on my story and my friends are talking about it too.

They're giving me a podcast you see the trend here.

They're taking the content and they're flipping it multiple times they're seeing that something works and instead of just moving on to the next thing they're taking that and doing it again and again and in new ways.

And so I'm sure we're all familiar with this your traditional marketing funnel.

But what I want to challenge you to do today based on what I've shared is think about how you can segment your audience is you may be thinking to yourself I work in take I work in life science I don't do something cool like netflix but this can really be applied to any industry even ignite we focus on life science we're focusing on architectures engineers construction and we have specific content as specific podcast and webinars and blogs and white papers and etcetera etcetera that speak just to them.

So in conclusion.

The landscape of marketing is ever evolving just when you find your sweet spot with marketing that means you need to get back out there learned some work because it's about to change on you again.

And segment, your audience you can't blanket your content you just can't you can try it if you want, but it's not going to stick.

And lastly your users want to be engaged with not and so what that means is listening to them and understanding what kind of content they want to receive very says settling them the concept that you think they want.

>> Now it's time for your questions for bashar black about how marketing is evolving, raise your hand we'll get to you with a question first question coming up from the front row here.

>> Hi, there I'm not sure that this is a direct to you but to all the folks who spoke I came here hoping to solve a problem that.

I'm experiencing.

I own a small company.

Literally got mothball during covid because we need to be outside.

We built it.

The infrastructure exists.

The principals of the company had to go find other things to do.

As covid seems too.

The waning we'd like to breathe life back into it and we know it needs digital marketing.

We don't know how to do what we don't know where to find a freelancer a small company or even a potential partner who want to do it for us so if anybody can help assured, appreciate it.

>> So a good question for you would be as people are trying to move into this digital marketing world, what things should they know with that should they be aware of.

>> So I think the first thing that you should know is that you always want to capture some type of information on people so that you can stay in contact with them weather is the e-mail or phone number or even as the mailing address that that's how you prefer to market, but you want to make sure that you can keep the conversation going and I actually think places like this the frontier where there are other small businesses is a great place to meet people because you can scale together and likely they'll have comparable rates that align with the size of your business.

>> All right other questions on digital marketing how it needs to evolve here in the post-pandemic era or I guess current pandemic era since I'm wearing a mask.

Okay question from social media, then the guy pulled out on my phone over there, you spoke about netflix kind of doing some micro targeting of finding people who could be a cohort and marketing to them with content do you see that as a direction people should go or is that very focused content marketing a springboard into larger more broad marketing.

>> So I definitely think that segmented audience is segment marketing is the way to go.

That's going to help build community and that's going to help it organically grow of course there's going to be contact content that applies across the board for your business, but ultimately you want to make sure that you have your segment audience is.

>> And followup question there with more and more digital marketing platforms becoming available all the time and people spending more time at home, what are the platforms that you think people need to be paying attention to as they bring out their digital marketing strategy.

>> So I know a lot of people were really into clubhouse a few months ago, I am indifferent on it.

I think the main thing is that whatever platform that you choose you do it.

Well, you know you don't want to spread yourself too thin trying to be on every platform it linked in is working for you and your audience is there the focus on linkedin and so on and so forth.

All right question from the back of the room here.

>> Well you kind of mentioned the very beginning about how people wore how it's kind of like no.

No 1, 5, 1, out just like the on a computer screen.

That's how I day to day life now so I guess is kind of want in life how has marketing had kind of have to change like this flip-flop with the whole pandemic like things really bad things are good again.

Things are really that again like how do you guys keep up with that unlike of all with that change.

It's a great question.

So I think you know of course when you're dealing with the public when we're in the midst of a pandemic something that can be emotional for different people in different ways you definitely want to be aware of when are good times to engage with your audience.

That's one.

But I also think someone touched on earlier just with everything being online is it's almost like a numbers game you want things to be high quality but you also want to be consistent consistently in front of your audience consistently, giving them opportunities to engage even like with the strong Blackley I'm MS. 4 or 5 post but that one post that's because I popped online while I was eating lunch, it was a great pose and maybe all 5 are good, but you know, I'm not catching every single one.

So the numbers getting which also wanted to be high quality.

>> And then last question what marketing innovation that has come out during the pandemic do you think is going to be the one that sticks around becomes the new normal for the next decade.

>> So something I've seen I think is really cool is people sending curated boxes and gifts to their clients their leaves their customers things that are getting them at home experiences that still make them feel tangibly attached to the brand.

So maybe that's like a swag bag or you know that's like a meal or a Y and there are some things so you're still getting something physical and you still still a bit physically connected.

But you're still safely at home.

>> All right, let's show a big hand for push are on the left.

So if you've got questions by digital marketing some of our folks will be hanging around after the show you can meet them and ask them any questions that you didn't get a chance to ask.

Also the Bucs yard will be open the bar will be opening back up so you can have a chance to hang around for a little bit longer.

But now we have come to our final speaker of the evening.

She's an assistant professor of marketing at Duke University and has a phd in computer science.

Please welcome doctor Alison J D hello everyone.

>> And yes I am in half months, pregnant, so I'm trying not to fall off the stage.

So tonight we're going to be talking about over the next personalization or just personalization because all personalization unless you have like 4 customers is run by an algorithm so this is often known as like the netflix problem in this case, I'm talking about books.

So these are 3 of my favorite books I like any criminal winter's tale east of Eden what book should I read next.

So this is again like the netflix problem and it's a really common problem any time you have lots of users and lots of items are lots of content to look at you can figure out.

What are the patterns there what users are liking what type of content and its it's so common that Google has a course on recommendation systems and that's the qr code that I want and if you have enough technical expertise you can take this course and implement an over them.

In your system and Google estimates of this course takes about 4 hours to complete so with that sufficient technical background and 4 hours of the school course you can implement this in your system.

And so why would you want to do this why would you want to create personalized recommendations, I'm sure you all have seen them on facebook and other media.

But no matter your business model recommendation systems can be really valuable.

So for instance, if you have a subscription based service then it can help a customer attention because you have your customers Data's and your competitor doesn't so then you can personalize their experience where your competitor cannot so that helps with customer retention if instead you are.

Retailer, that make a purchase based model then you can increase the order value.

So you know add extra items to market basket.

By recommending you know you might also like this.

So then under that business model that also helps.

And last if you are in engagement based model of revenue.

It's going to keep people on your system or on your platform, because they're going to be more engaged.

They're going to get content that's more relevant to them and so you're going to get more and clicks so whatever your business model as long as it's you know.

I mean it doesn't even have to be online can be in person.

Recommendation systems can be really really valuable.

We talk about the benefits of a son and my kind of alluded to it as well.

But there's a problem with the systems and the problem is the data and in particular if you're how platform you make recommendations and then the users are clicking on those recommendations are there purchasing are engaging with those recommendations.

What are we doing with that bit of that and it's getting fed back into your platform, you're using that data to then make new recommendations moving forward.

So there's a feedback loop here.

This is what I research a research you know what what can go wrong when we have a feedback loop like this.

So I just wanted to talk about.

What are the questions you need to be asking yourself or your data scientists as you're implementing the systems to minimize negative impacts and to minimize the feedback with the facts.

The first question is the absolute most important question which is what valuation metrics that he is so if you're doing anything online.

I highly recommend this book that I've linked in the lower right which is trustworthy online controlled experiments.

And in this book is fantastic because it talks of it in great detail about what metrics align with your business and so whatever metrics are using to kind of capture.

What's valuable to use a business that's what you should be using in your experience to evaluate your recommendation systems and if you have a undergrad in computer science who's you know has taken this 4 hour course they might not know what's valuable to your business right.

So you need to think as either a business owner or manager about what's important to you what do you want to improve and then how do you evaluate your system kind of in alignment with that all right and so then these are the questions are.

You know just things I've come across is, I've talked to the company says they've implemented the system.

And the second one shocks me released their answers shocking so the second question is when users recommended something and they don't engage with it.

They don't click on it.

They don't buy it are you storing that information and most of the time firms are like no.

No and it's you know we start with a week and this is crucial because what they don't like is just as important as what they do like because you don't want to provide them the things they don't like.

So when when possible and it is possible because data storage is cheap store what they don't like worst or what they don't click on because that's very informative.

How do you treat new users and items.

Also knowing crucially important.

There are lots of good ways to do it and there are lots of bad ways to do it so again these are just conversation points.

The need to have with your data scientists or with yourself if you're doing it yourself in order to ensure that your recommendation system is kind of pushing years is in the right direction.

And again the metrics the valuation metrics are hugely important because that kind of can catch some of the errors that well, you know come forth because of these other questions if you didn't do things right.

You know how frequently area updating the model of frequently updating, you know with new data and then what other constraints to consider because of a naive recommendation system will just kind of give you what it thinks you like best so say like star Wars, you know what would happen if I get a recommendations that are I like star Wars and more star Wars and more star Wars and more star Wars like it's not useful.

So you need to include other things in order to ensure that your recommendation systems are diverse and you know novel to users she need to be thinking carefully not just about the algorithms but again the whole business model as you're getting as your integrating this with your platform, so I just want to leave with that and I'm happy to answer questions about either recommendation systems or no machine learning more broadly since that's kind of my area of expertise some.

Thank you.

>> You've seen it you've interacted with them now you can ask questions about the recommendation systems what's on your mind razor.

>> And and we will bring the microphone out to you to get your questions got one in the back here kind of around the corner for use.

I'm going to go this way.

Why my way through the halls of the front here all right first question.

>> It's some is correct.

So.

>> How would that and so I guess one thing that you sort of touched on it the last ball point is how would this work with difference.

Region the world and different privacy in such an.

I worked at one company where the recommendations that worked really well, but the companies concerned that gpr such so for a European use as we just heard from generic thing so you know, I guess like how do you kind of you know balance warnings give good recommendations but also to switching over cautious lawyers.

>> Absolutely so there are some privacy preserving recommendation systems and part of that is you introduce random missing to your training data.

So that it.

The users are kind of smoothed out so you can think of recommendation systems as you know really fine segmentation right and so you're kind of it's the needle you're pushing.

Are you are you sort of a core segmentation or you have or an individual level segmentation and if you can push the needle a little bit towards the course or segmentation than that the privacy preserving and it's not unique to every user so then you're just kind of taking a hybrid approach there.

And that's that's kind of the best technique that you can do to to comply with things like gdpr.

>> Next question is over here to your right.

So, how do you think about creating these how do you how do you think about creating trust and in these algorithms with either with your customers or business.

>> So the question is about creating trust because this can definitely have a little bit of a big brother feel to it.

You know part of that is again addressing some of the GDR gdpr issues that's one of the reasons gdpr came about is to protect users.

So it also comes back to that last question which is we know what else are you including in building.

The recommendation systems because part of it is you want to recommend some things that are predictable that you want to recommend you want to recommend a star Wars movie perhaps because then I can look at that list and I know I like star Wars, therefore, you know maybe like these other things in the list, but you don't want to recommend only what's predictable because then it's not useful so it's this balance of what are all the considerations that you want to include, including you know a little bit of predictability a little bit of unpredictability so that the users are pleasantly surprised in some ways is but not completely surprised and so it's just adding enough noise to the system.

So that.

Again like the surprise but not totally surprised it it's a hard balance, but it's kind of you know, you know you see it.

>> Next question coming in from the front row from one of our speakers.

>> So I'm just wondering if you're maybe a startup or nonprofit, which of those questions would you focus on for you don't you don't have a data scientist on staff or you don't have you know what's the most important thing you want to focus on.

>> It's what matters he is and it's absolute moment are seizing up.

Because you know you're going to be the best way to do is to use some sort of a B testing to figure out what works best for you and so what metric are using maybe test is it clicks probably not I'm just because that creates click way that doesn't create long-term engagement, you want to think about you know measuring user activity or something you no longer term that aligns well with your business metrics motivations next question coming into us from the back room here.

>> Yes, so I had a question about modeling in general, so like even now are becoming more connected globally.

I guess so previous models might have had is a slightly smaller scale.

I'm learning how it modeling changes in general as more connections happen more.

If you have more cell is like people to buy a better buy your product.

>> So so these algorithms scale really really well.

We're talking millions of users millions of items in fact the don't work.

Well when you only have like a 100 users or a 100 I don't think it's that tends to be too small and that you have a really dense matrix and so in terms of scale.

These are built for scale these are built for the big things in the trickier part is actually making them work in a small scale.

>> And the last question coming in from social media using the hashtag RDP one 80 person says I bought a dishwasher now all I see is ads for dishwashers how many dishwashers do they think I need.

>> This is kind of a question that was hinted at before and I think that's just a mistake.

Honestly I think that's a failure to communicate between the purchase pipeline and the recommendation system because usually those are separated for privacy reasons and so as we you know create integrated models that kind of include as much information as possible not just what we clicked on right because if that if we're getting a recommendation for something that you've already purchased the model is probably trained exclusively on what you've been looking at and so if we integrate what we actually purchased as well as what we are looking at that you can create better recommendations.

Lazio a big damper doctor Allison.

>> Casey ladies and gentlemen, we did it.

We didn't RDP one 80 in person.

Thank each and every one of you for making it out here tonight up next in September food Syan to food science.

The science of food.

A lot of 5 speakers begin on the subject of food science fingers crossed we'll be right back here in the front here September 16 05:00PM to 07:00PM streaming live on PBS in C don't forget to the triangle pop-up market at the Bucs are coming up here on Saturday.

Some of our speakers are sticking around you can talk to them the taps will be opening back up for a little while the backyard right over there.

Enjoy yourselves of being out of the House on behalf of RDP one 80 presented by rti international I've been your see wait matter thank you very much for coming out and have a safe journey home will see you next month.