JeffGreenberg

easy way to remove these vertical bar artifacts?

Recommended Posts

M.Chapman    519
8 hours ago, wiskerke said:

 

Is this from your SD card? It looks suspiciously like blinkies for over exposure.

Do you have one or two directly from your SD card, that have not gone through some sort of processing yet?

 

The fact that those stripes are not parallel is baffling. I have never seen anything like it.

 

wim

Wim,

 

I've downloaded the DNG.  I believe the  stripes would be parallel if we could see the raw sensor data before automatic lens distortion and CA correction that has been applied. If I apply -37 manual lens distortion correction in LR (in an attempt to cancel the correction already applied) the bars become vertical and parallel. They are still slightly "smeared" due to the double distortion correction. I think, if we could see the original data they would be perfect 1 pixel wide dark vertical lines that occur only in clipped highlight areas. Looks like a sensor or data processing problem in the camera to me, and nothing to do with the SD card.

 

Jeff,

 

The DNG file you posted looks like it was produced in LR or with ACR (it has all sorts of corrections applied highlights, vibrance, etc.) do you have a *real* RAW file from the camera that hasn't been processed? If you have it's sometimes possible to turn off distortion correction (by editing the EXIF settings) or by using DXO optics to convert the RAW. This would then leave the dark bard in their "purist" form, potentially making them much easier to remove, for example using the PS automatic dust and scratch filter with high threshold.

 

Mark

 

 

Edited by M.Chapman
  • Upvote 1

Share this post


Link to post
Share on other sites
wiskerke    1,842
4 hours ago, M.Chapman said:

Wim,

 

I've downloaded the DNG.  I believe the  stripes would be parallel if we could see the raw sensor data before automatic lens distortion and CA correction that has been applied. If I apply -37 manual lens distortion correction in LR (in an attempt to cancel the correction already applied) the bars become vertical and parallel. They are still slightly "smeared" due to the double distortion correction. I think, if we could see the original data they would be perfect 1 pixel wide dark vertical lines that occur only in clipped highlight areas. Looks like a sensor or data processing problem in the camera to me, and nothing to do with the SD card.

 

Jeff,

 

The DNG file you posted looks like it was produced in LR or with ACR (it has all sorts of corrections applied highlights, vibrance, etc.) do you have a *real* RAW file from the camera that hasn't been processed? If you have it's sometimes possible to turn off distortion correction (by editing the EXIF settings) or by using DXO optics to convert the RAW. This would then leave the dark bard in their "purist" form, potentially making them much easier to remove, for example using the PS automatic dust and scratch filter with high threshold.

 

Mark

 

 

 

According to the exif the correction was not done by ACR.

If you open the dng file in ACR, all settings are readily available. (And reversible.)  If you open it in a editor or even when you look at it in your browser, all exif headers appear at the front. However there's no way to tell the difference between the corrections ACR made and the ones written by DJI. Except that we know that ACR does not write stuff about a gimbal.

 

Agree we would need true RAW images and how the images were obtained to at least understand what the problem is.

Settings would be helpful too. Like HDR or not; AEB; synchronize files; sharpening; 3D noise (what 3D?) and so on.

Maybe it is in the EXIF, it's just that I don't understand what I'm looking at.

 

In the mean time I would quickly test a different card; different card reader; different settings; camera without drone or gimbal (probably impossible or not practical). Just to rule all that out.

 

wim

  • Upvote 1

Share this post


Link to post
Share on other sites
JeffGreenberg    358

Thanks for further responses.

Yes, sample DNG has CS6 RAW adjustments.

But those can be reversed as Wim says...?

 

"This would then leave the dark bard in their "purist" form, potentially making them much easier to remove, for example using the PS automatic dust and scratch filter with high threshold."

 

Does auto dust filter make use of "content aware", similar to PatchTool?

Works entire image rather than what one surrounds with tool?

Yes, that could be quicker...

Is it offered in CS6?

 

Share this post


Link to post
Share on other sites
M.Chapman    519
15 hours ago, JeffGreenberg said:

Thanks for further responses.

Yes, sample DNG has CS6 RAW adjustments.

But those can be reversed as Wim says...?

 

"This would then leave the dark bard in their "purist" form, potentially making them much easier to remove, for example using the PS automatic dust and scratch filter with high threshold."

 

Does auto dust filter make use of "content aware", similar to PatchTool?

Works entire image rather than what one surrounds with tool?

Yes, that could be quicker...

Is it offered in CS6?

 

 

Hi Jeff,

 

If you open the unprocessed RAW image in CS6 then yes, you might* be able to reverse the automatic distortion correction by adjusting using the lens distortion correction tab and dragging the slider back towards barrel distortion (I found -37 was about right). Try to get the black bars near the corner of the frame to turn back into very narrow (ideally single pixel) perfectly vertical stripes. (It may not be possible to get single pixel width, depending on where the fault is occurring and the demosaic algorithm)

 

*There's no guarantee you'll be able to get a perfect cancellation of the automatic distortion correction since the automatic correction may use a more complex distortion adjustment (based on parameters in the raw file, or in Adobe's lens profile model). You could try turning the any automatic lens distortion correction off, if that option is available. But with my camera that requires editing the EXIF as that's were the parameters are stored.

 

If you can get a good cancellation so the bars are only a pixel wide and not smeared, then try the noise and dust removal filter in CS6 (I think it's got the same filter PSE has). Try Filter > Noise > Dust &Scratches. Suggest setting a high threshold and the radius as small as you can get away with. On PSE it removes dust and scratches and does a content aware fill, and yes, it works on the entire image. But.... be careful to check that other narrow dark objects (telegraph poles and wires etc.) haven't also been removed!  You may want to use in combination with a selection brush to avoid collateral damage.

 

Again, if you can post an unprocessed RAW image, I'm happy to experiment.

 

Mark

 

 

Edited by M.Chapman
  • Upvote 1

Share this post


Link to post
Share on other sites
M.Chapman    519

Hi Jeff,

 

I did some more playing with the DNG and got some encouraging results.

 

1) Remove lens distortion correction in LR so that the bars all appear vertical (and max 2 pixels wide). Picture looks a bit "fish eye", drone camera lens obviously suffers from massive barrel distortion. :-(

2) Export to Photoshop as PSD or TIFF

3) In PS. Image resize to exactly 50% of width and height in pixels using nearest neighbour algorithm (makes bars only 1 pixel wide)

4) Duplicate background layer (becomes top layer)

5) Apply Custom filter 1  -1   1 with scale 1, offset -10 to top layer. This will turn the high contrast vertical black bars white. (It's basically an invert and sharpen filter for vertical edges).

6) Apply threshold of 249 to new layer (so <249 = black and >249 = white)

7) Select all white areas on new layer (magic wand selection tool, tolerance 1, non-contiguous). This effectively makes a mask.

8) Hide top layer and select background layer to work on 

9) Hide selection mask outline (cntrl H) so you can see what's going on

10) Apply Filter>Noise>Dust and scratches radius 2 threshold 1. Boom! the bars pretty much disappear

11) Clear selection (deselect)

12) Delete top layer

13) Reapply distortion correction

14) Upsize image to meet minimum Alamy size requirement (17MB).

 

Maybe the above can be stored as a PS action?

 

NB. Although it's a partially automatic process it's still far from perfect and would still fail QC without some manual cloning.

It might work better if starting from a true RAW image.

 

Here's a screenshot before (right) and after (left) step 10.

 

Screen_Shot_2017-08-05_at_20.53.18.jpg

Mark

 

 

 

  • Upvote 3

Share this post


Link to post
Share on other sites
JeffGreenberg    358

MarkC & all,

 

Amazing results!

Of those (14) steps, (10) of them I know nothing about!

Here is link to unopened DNG shot today, so am hoping all data is P4P+ specific...

 

www.photosphotos.net/DJI_0118.DNG

 

change ".DNG" in URL to ".JPG" to get untouched JPG shot simultaneously;

was hoping JPG might be artifact-free but...

FYI, blow up JPG to 3200% & see artifacts are not just 2 pixels wide,

but there are also very faint additional vertical bars...

 

Edited by JeffGreenberg
because it seemed like the right thing to do

Share this post


Link to post
Share on other sites
M.Chapman    519
22 hours ago, JeffGreenberg said:

MarkC & all,

 

Amazing results!

Of those (14) steps, (10) of them I know nothing about!

Here is link to unopened DNG shot today, so am hoping all data is P4P+ specific...

 

www.photosphotos.net/DJI_0118.DNG

 

change ".DNG" in URL to ".JPG" to get untouched JPG shot simultaneously;

was hoping JPG might be artifact-free but...

FYI, blow up JPG to 3200% & see artifacts are not just 2 pixels wide,

but there are also very faint additional vertical bars...

 

Hi Jeff,

 

I found an easier fix for the defect bands.

 

I looked at your latest (RAW) DNG and it's now clear that lens distortion correction is being applied by ACR or LR when I open the file. LR says

This raw file contains a built-in lens profile for correcting distortion, chromatic aberration and vignetting. The profile has already been applied automatically to this image.

This cannot easily be turned OFF in LR or ACR.

 

So, I downloaded On1 RAW (a dumber image editor) which doesn't carry out automatic lens distortion and opened your DNG in that.

 

Bingo. Image distortion correction is no longer applied. The image shows loads of vignetting and barrel distortion (it's a pretty poor lens/camera), but now the defect bands are perfectly vertical and only 1 pixel wide (if on white background) and (where they appear) they only occur on every 10th column of pixels. So now all we need to do is throw away those columns of pixels.

 

I used On 1 to export as a 16 bit PSD and then opened in PS. I then used Image>Resize>Pixels 50%* using nearest neighbour and the defect bars are completely (and perfectly) eliminated! The image now needs fixing for vignetting and distortion, but that's not too tricky. It will also need slight upsizing to meet minimum Alamy size (17MB)

 

*In theory it should be possible to downsize by just 10%, but I had trouble synchronising the columns of pixels that are discarded with the defect bars. I'll investigate this further.

 

Mark

Edited by M.Chapman
  • Upvote 2

Share this post


Link to post
Share on other sites
M.Chapman    519

Hi Jeff,

 

A minor refinement that allows downsizing by just 10% (instead of 50%).

 

After opening the PSD* in Photoshop, do the following;

 

1) Use Image>Resize>Canvas Size... to chop 4 pixels off the right-hand edge of a horizontal format image so the image becomes 5,460 pixels wide. i.e. divides exactly by 10

2) Use Image>Resize>Image Size... to reduce pixel size to 90% using nearest neighbour algorithm (this will throw away 10% of the pixels (1 in 10 columns and rows are discarded. The PSE 90% resize removes pixels in the 5th, 15th, 25th etc... columns, and luckily these are exactly the columns which contain the dark band defects.

3) Note that some white defects remain - look at the distant building skyline. So some cloning may still be needed.

 

The above works in PSE, I hope it's the same in CS6.

 

Here's before (right) and after (left) 100% crops from the latest DNG you posted showing almost perfect removal with just a 10% downsize. 

 

Screen_Shot_2017-08-06_at_17.58.10.jpg

 

*I used On 1 to create the PSD file without any lens distortion correction from your DNG. It's possible other RAW convertors may work, but I got the best result with On 1. If you do use a different convertor the columns that the defect bars appear in may shift by +/-1 pixel, so the resize canvas process would need altering.

 

I did some checking to see where the lens distortion correction data used by ACR/LR is stored. It's not an Adobe profile, it's supplied by the camera manufacturer and is stored inside the DNG file, hidden in the OpCodeList parameters. I don't have any tools to edit these, but did find an app (for Mac) which can remove them, see https://itunes.apple.com/us/app/dng-cleaner/id1205138449?mt=12 . When I tried it, it worked fine and when I opened the file in LR the lens correction was removed, but defective pixel bands weren't pure 1 pixel wide. There was some weird "bleeding" into adjacent pixels (see below) probably due to a different demosaic algorithm in LR. So that's no good, which is a shame, as it would have provided a fix that didn't need another RAW convertor.

 

Screen_Shot_2017-08-06_at_18.18.07.jpg

 

PS. Interestingly it looks like On 1's RAW convertor is quite good. A good balance between detail and noise which I struggled to beat in LR.

 

Mark

 

Edited by M.Chapman
  • Upvote 2

Share this post


Link to post
Share on other sites
wiskerke    1,842

DJI by now must know the problem and have a solution, because many people have been complaining about this since at least 2015.

There are complaints from people with a p2 already. Tons of complaints from people with p3p's.

Google phantom camera zebra stripes (yes most people understand they're not true 45 degree zebra stripes) or phantom camera vertical black lines.

Most suggest ribbon cable and harness as source of problems.

Youtube 2015 - Someone in reactions claims solution.

Others urge to get the thing back to DJI asap (because of short warranty period?)

DJI is not the only one. There's a Yuneec owner with the same problem.

 

wim

 

  • Upvote 1

Share this post


Link to post
Share on other sites
M.Chapman    519
3 hours ago, wiskerke said:

DJI by now must know the problem and have a solution, because many people have been complaining about this since at least 2015.

There are complaints from people with a p2 already. Tons of complaints from people with p3p's.

Google phantom camera zebra stripes (yes most people understand they're not true 45 degree zebra stripes) or phantom camera vertical black lines.

Most suggest ribbon cable and harness as source of problems.

Youtube 2015 - Someone in reactions claims solution.

Others urge to get the thing back to DJI asap (because of short warranty period?)

DJI is not the only one. There's a Yuneec owner with the same problem.

 

wim

 

 

Interesting. I note that there's also quite a lot of discussion about over-exposure warning stripes too. Looking at the images above I note that the stripes are misplaced to the right, relative to the highly exposed areas. That's weird... If the problem was in the sensor (e.g. overflow) shouldn't it be more perfectly synchronised with the light areas? Is it possible it's a rogue bit of "over-exposure warning" code in the firmware that's leaving its footprint?

 

Mark

  • Upvote 1

Share this post


Link to post
Share on other sites
JeffGreenberg    358
MarkC-Wim & all,
 
Briefly, cable-band btwn body & camera as issue rings true!
Wim, will pass those links on to DJI when reporting my issue...
FYI, my OverexposureWarning function was always OFF...

(73) P4P+ have just passed QC (~20 fixed via manual PatchTool)!!!
 
Have determined how to avoid artifacts!
(at least until I send to factory for repair under warranty)
Under 1/3rd of (73) recent keepers had artifacts, so had hunch there'd be avoidance strategy.
First thought it was sun angle, shooting into sun vs. sun behind.
But its much bloody simpler -- parallel bar artifacts lay on clipped-overexposed areas!!!
Like my RX10-III, my P4P+ was set on +0.7 exposure.
By shooting same composition, same lighting, over & under exposed, it revealed truth, e.g.:
 
(lots of artifacts in sky, way over exposed clouds)
 
(almost no artifacts, just a few PatchTool manageable upper left edge)
 
Now I can take P4P+ on (2) pending regional trips with confidence
& send to factory afterwards;  Am always shoot at ISO 200 & f4.5 in sun,
so underexposure can be easily corrected without noise issues...?
 
MarkC:  am feeling guilty about all you did & me not being able to make use
due to my processing skill limitations.  Other than giving you greenies
every time I see your posts on any topic, I owe you scones with jam
& tea if I ever figure out how to fly award seat US<==>UK without having
to pay UK huge fuel surcharges; fee avoidance involves landing in
Ireland or Europe first, I think, then on to UK...regards jg
Edited by JeffGreenberg

Share this post


Link to post
Share on other sites
M.Chapman    519
2 hours ago, JeffGreenberg said:
MarkC:  am feeling guilty about all you did & me not being able to make use
due to my processing skill limitations.  Other than giving you greenies
every time I see your posts on any topic, I owe you scones with jam
& tea if I ever figure out how to fly award seat US<==>UK without having
to pay UK huge fuel surcharges; fee avoidance involves landing in
Ireland or Europe first, I think, then on to UK...regards jg

 

Hi Jeff

 

No need to feel guilty. I took it as a challenge to see if those bars could be removed, and learnt quite a bit on the way. The app I found that removes lens correction data will also be quite useful to me as it recovers so many pixels which can often be used in shots that don't contain architectural details. I'll look forward to the scones, jam and tea. I might even bring some cream ;)

 

Mark

  • Upvote 1

Share this post


Link to post
Share on other sites
wiskerke    1,842

I will be in the US most of September. Not in Miami though. (Roughly DC>Salt Lake>SF)

Scones will have to wait. Prefer BBQ anyway.

 

wim

  • Upvote 1

Share this post


Link to post
Share on other sites
JeffGreenberg    358
3 hours ago, wiskerke said:

(Roughly DC>Salt Lake>SF)

wim

 

Wild itinerary!  Almost like visiting 3 countries...

Hopping by air?  I-70 + I-80 one way rental...?

regards jg

Share this post


Link to post
Share on other sites
wiskerke    1,842
2 hours ago, JeffGreenberg said:

 

Wild itinerary!  Almost like visiting 3 countries...

Hopping by air?  I-70 + I-80 one way rental...?

regards jg

 

DC-Salt Lake by air.

Not sure if I am going to reach SF at all, but that's the general direction, by car. Somehow not 1 way, but that may change.

Why? Just haven't been in those parts or not long enough.

Salt Lake because we have to be there for a couple of days. DC because we want to be there a couple of days. ;-)

The car will probably be 4wd and I will be bringing my tent and my coffee maker.

There are no camp sites in the US without a BBQ grill.

All of this trip is after Labor Day: camp sites will be nice and quiet.

I'm not much of a landscape photographer, that's the only thing against it. And why I may throw in one or two cities.

 

wim

Share this post


Link to post
Share on other sites
wiskerke    1,842
2 hours ago, Cecile Marion said:

Too bad, DC>Salt Lake>SF are not particularly known for their BBQ. 😉

 

Yes that crossed my mind also. However: there are no camp sites in the US without a BBQ grill.

And places in the US not known for good BBQ, may still be way better than anything over here.

Then again, have you ever tasted a good parilla or asado? Owww.

 

wim

Share this post


Link to post
Share on other sites
JeffGreenberg    358

Although BBQ may not be widespread in DC-SLC-SF,

I guarantee, without even Googling, there will be one or a
few best BBQ places in each city -- & they will be really good...

Share this post


Link to post
Share on other sites
Johnnie5    185

Wim,

 

September is a great time to visit Yosemite and if you hang a left at Reno onto highway 395 you can see some of that Ansel Adams territory on the eastern side of the Sierra Nevada Mountains.  One of the most pleasant detours you will ever take and plenty of uncrowded campgrounds that time of year.  Drive over the mountains on Highway 120 through Tioga Pass which this year has only been snow free and open since the end of June and Mammoth mountains ski area didn't shut down till last week.

Share this post


Link to post
Share on other sites
wiskerke    1,842
7 minutes ago, Johnnie5 said:

Wim,

 

September is a great time to visit Yosemite and if you hang a left at Reno onto highway 395 you can see some of that Ansel Adams territory on the eastern side of the Sierra Nevada Mountains.  One of the most pleasant detours you will ever take and plenty of uncrowded campgrounds that time of year.  Drive over the mountains on Highway 120 through Tioga Pass which this year has only been snow free and open since the end of June and Mammoth mountains ski area didn't shut down till last week.

 

Duly noted! Thank you!

That may well be the route I will be taking. That or the other way around.

 

wim

Share this post


Link to post
Share on other sites
Betty LaRue    1,053

Too bad you can't hit Kansas City, Mo., known for BBQ. It's off your beaten path, though, but if you weren't flying from D.C. and traveling by car, doable. Narrowing down the BBQ restaurants would be the hard part.

 

i remember when we went to San Francisco the first time. We walked Fisherman's Wharf. The eateries had menus outside their doors. We looked, studied, finally picked one. Ordered platters that had a bit of everything.  And everything was dipped in the same batter and fried. It all tasted the same and wasn't good and it was overpriced. Don't eat at Fisherman's Wharf!

 

We traveled in our van to Washington state and got into a KOA campground a block from the sea at dusk a few days later. Asked the proprietor where we could eat. She directed us to a place a couple of streets away. 

Inside the door was a narrow room with a fresh fish counter at one end, 3 tiny tables at the other end. Through the door to the back was a huge room with pool tables. Fishermen unwinding after a day on the boats and locals playing pool and laughing. We could smell the beer fumes.

Our dinner was the best seafood dinner I've ever eaten in my life, including the oyster stuffing served on a seashell. Ferreting out those places are usually strokes of luck. You just stumble upon them.

Betty

Share this post


Link to post
Share on other sites
JeffGreenberg    358
wiskerke    1,842
6 hours ago, JeffGreenberg said:

 

Great mouth watering article!

Why did we have to go back to Holland ;-)

 

wim

 

edit: typo

Edited by wiskerke

Share this post


Link to post
Share on other sites
wiskerke    1,842
1 hour ago, Johnnie5 said:

If you happen to get to the Reno-Sparks area before the 4th of September there is the Nugget Rib cook-off.  http://www.nuggetribcookoff.com/

 

OMG.

Pity, no.

 

5 pages %bbq% on AoA.

15 for %salad%.

Maybe I should start taking pictures of food.

 

wim

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now